aboutsummaryrefslogtreecommitdiff
path: root/src/runtime/internal/atomic
AgeCommit message (Collapse)Author
2022-12-17cmd/compile: sign-extend the 2nd argument of the LoweredAtomicCas32 on ↵Guoqi Chen
loong64,mips64x,riscv64 The function LoweredAtomicCas32 is implemented using the LL-SC instruction pair on loong64, mips64x, riscv64. However,the LL instruction on loong64, mips64x, riscv64 is sign-extended, so it is necessary to sign-extend the 2nd parameter "old" of the LoweredAtomicCas32, so that the instruction BNE after LL can get the desired result. The function prototype of LoweredAtomicCas32 in golang: func Cas32(ptr *uint32, old, new uint32) bool When using an intrinsify implementation: case 1: (*ptr) <= 0x80000000 && old < 0x80000000 E.g: (*ptr) = 0x7FFFFFFF, old = Rarg1= 0x7FFFFFFF After run the instruction "LL (Rarg0), Rtmp": Rtmp = 0x7FFFFFFF Rtmp ! = Rarg1(old) is false, the result we expect case 2: (*ptr) >= 0x80000000 && old >= 0x80000000 E.g: (*ptr) = 0x80000000, old = Rarg1= 0x80000000 After run the instruction "LL (Rarg0), Rtmp": Rtmp = 0xFFFFFFFF_80000000 Rtmp ! = Rarg1(old) is true, which we do not expect When using an non-intrinsify implementation: Because Rarg1 is loaded from the stack using sign-extended instructions ld.w, the situation described in Case 2 above does not occur Benchmarks on linux/loong64: name old time/op new time/op delta Cas 50.0ns ± 0% 50.1ns ± 0% ~ (p=1.000 n=1+1) Cas64 50.0ns ± 0% 50.1ns ± 0% ~ (p=1.000 n=1+1) Cas-4 56.0ns ± 0% 56.0ns ± 0% ~ (p=1.000 n=1+1) Cas64-4 56.0ns ± 0% 56.0ns ± 0% ~ (p=1.000 n=1+1) Benchmarks on Loongson 3A4000 (GOARCH=mips64le, 1.8GHz) name old time/op new time/op delta Cas 70.4ns ± 0% 70.3ns ± 0% ~ (p=1.000 n=1+1) Cas64 70.7ns ± 0% 70.6ns ± 0% ~ (p=1.000 n=1+1) Cas-4 81.1ns ± 0% 80.8ns ± 0% ~ (p=1.000 n=1+1) Cas64-4 80.9ns ± 0% 80.9ns ± 0% ~ (p=1.000 n=1+1) Fixes #57282 Change-Id: I190a7fc648023b15fa392f7fdda5ac18c1561bac Reviewed-on: https://go-review.googlesource.com/c/go/+/457135 Run-TryBot: Than McIntosh <thanm@google.com> Reviewed-by: Cherry Mui <cherryyz@google.com> Reviewed-by: Matthew Dempsky <mdempsky@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Wayne Zuo <wdvxdr@golangcn.org> Reviewed-by: Than McIntosh <thanm@google.com> Reviewed-by: David Chase <drchase@google.com>
2022-10-29runtime/internal/atomic: enable atomic 64bit types for all functions on ↵Guoqi Chen
linux/loong64 ref. CL 356169 Change-Id: I813750d3fe191afe50d4029024ba97800211f688 Reviewed-on: https://go-review.googlesource.com/c/go/+/414414 Reviewed-by: WANG Xuerui <git@xen0n.name> Reviewed-by: David Chase <drchase@google.com> Run-TryBot: Wayne Zuo <wdvxdr@golangcn.org> Reviewed-by: Heschi Kreinick <heschi@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-10-18runtime/internal/atomic: add write barrier-enabled pointer atomicsRuss Cox
UnsafePointer.Store, UnsafePointer.CompareAndSwap were missing, although .StoreNoWB and .CompareAndSwapNoWB existed. Same for Pointer[T}. Do the linkname tricks necessary to add those methods. Change-Id: I925ee27673288accb15ebe93898f9eb01ab46a98 Reviewed-on: https://go-review.googlesource.com/c/go/+/443379 Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Austin Clements <austin@google.com> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-08-26runtime: mark all methods of atomic types with go:nosplitAndy Pan
Atomic types may be used anywhere in the runtime, so they must omit its usual stack overflow checks to avoid errors from stack splits. Change-Id: Icb497334c860bcaa284a2d5a3edc47ee89844301 Reviewed-on: https://go-review.googlesource.com/c/go/+/425484 Reviewed-by: Cuong Manh Le <cuong.manhle.vn@gmail.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> Auto-Submit: Cuong Manh Le <cuong.manhle.vn@gmail.com> Reviewed-by: David Chase <drchase@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com>
2022-08-19runtime/internal/atomic: remove double-check in kernelcas on linux/armTobias Klauser
Older kernels which require the double check of the __kuser_cmpxchg result are no longer supported as of Go 1.18 which requires at least Linux 2.6.32. For #45964 Change-Id: Ic3d6691bf006353ac51b9d43e742d970e3d4e961 Reviewed-on: https://go-review.googlesource.com/c/go/+/424556 Auto-Submit: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> Reviewed-by: Cherry Mui <cherryyz@google.com>
2022-08-12runtime: mark atomic methods which call nosplit functions as nosplitCuong Manh Le
Fixes #54411 Change-Id: I482ebca7365862bfb82a9daf8111c6f395aa1170 Reviewed-on: https://go-review.googlesource.com/c/go/+/423255 Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-08-08runtime/internal/atomic: add Pointer[T] typeAustin Clements
Change-Id: If8fcb37f4a8fcc0668af0df12f1cb8c66f2d2eea Reviewed-on: https://go-review.googlesource.com/c/go/+/418954 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-05-19runtime/internal/atomic: add atomic support for loong64Xiaodong Liu
Contributors to the loong64 port are: Weining Lu <luweining@loongson.cn> Lei Wang <wanglei@loongson.cn> Lingqin Gong <gonglingqin@loongson.cn> Xiaolin Zhao <zhaoxiaolin@loongson.cn> Meidan Li <limeidan@loongson.cn> Xiaojuan Zhai <zhaixiaojuan@loongson.cn> Qiyuan Pu <puqiyuan@loongson.cn> Guoqi Chen <chenguoqi@loongson.cn> This port has been updated to Go 1.15.6: https://github.com/loongson/go Updates #46229 Change-Id: I0333503db044c6f39df2d7f8d9dff213b1361d6c Reviewed-on: https://go-review.googlesource.com/c/go/+/342320 Reviewed-by: David Chase <drchase@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com>
2022-05-13runtime/internal/atomic: align 64-bit types to 8 bytes everywhereMichael Anthony Knyszek
This change extends https://go.dev/cl/381317 to the runtime/internal/atomic package in terms of aligning 64-bit types to 8 bytes, even on 32-bit platforms. Change-Id: Id8c45577d07b256e3144d88b31f201264295cfcd Reviewed-on: https://go-review.googlesource.com/c/go/+/404096 Reviewed-by: David Chase <drchase@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Knyszek <mknyszek@google.com>
2022-04-05all: separate doc comment from //go: directivesRuss Cox
A future change to gofmt will rewrite // Doc comment. //go:foo to // Doc comment. // //go:foo Apply that change preemptively to all comments (not necessarily just doc comments). For #51082. Change-Id: Iffe0285418d1e79d34526af3520b415a12203ca9 Reviewed-on: https://go-review.googlesource.com/c/go/+/384260 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2022-03-31runtime/internal/atomic: add BoolMichael Anthony Knyszek
This change adds the Bool type, a convenient wrapper around Uint8 for atomic bool values. Change-Id: I86127d6f213b730d6999db5718ca1a5af0c5b538 Reviewed-on: https://go-review.googlesource.com/c/go/+/393395 Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Austin Clements <austin@google.com> Trust: Michael Knyszek <mknyszek@google.com>
2022-02-25runtime/internal/atomic: set SP delta correctly for 64-bit atomic functions ↵Cherry Mui
on ARM 64-bit atomic functions on ARM have the following structure: - check if the address is 64-bit aligned, if not, prepare a frame and call panicUnaligned - tail call armXXX or goXXX depending on GOARM The alignment check calls panicUnaligned after preparing a frame, so the stack can be unwound. The call never returns, so the SP is not set back. However, the assembler assigns SP delta following the instruction stream order, not the control flow. So it leaves a nonzero SP delta after the check, to the tail call instructions, which is wrong because when they are executed the SP is not decremented. This CL fixes this by adding the SP back (the instruction never executes, just tells the assembler to set the SP delta back). Should fix #51353. Change-Id: I976cb1cfb0e9008b13538765cbc7eea0c19c7130 Reviewed-on: https://go-review.googlesource.com/c/go/+/388014 Trust: Cherry Mui <cherryyz@google.com> Run-TryBot: Cherry Mui <cherryyz@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com>
2021-12-13all: gofmt -w -r 'interface{} -> any' srcRuss Cox
And then revert the bootstrap cmd directories and certain testdata. And adjust tests as needed. Not reverting the changes in std that are bootstrapped, because some of those changes would appear in API docs, and we want to use any consistently. Instead, rewrite 'any' to 'interface{}' in cmd/dist for those directories when preparing the bootstrap copy. A few files changed as a result of running gofmt -w not because of interface{} -> any but because they hadn't been updated for the new //go:build lines. Fixes #49884. Change-Id: Ie8045cba995f65bd79c694ec77a1b3d1fe01bb09 Reviewed-on: https://go-review.googlesource.com/c/go/+/368254 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Robert Griesemer <gri@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
2021-11-06all: remove more leftover // +build linesTobias Klauser
CL 344955 and CL 359476 removed almost all // +build lines, but leaving some assembly files and generating scripts. Also, some files were added with // +build lines after CL 359476 was merged. Remove these or rename files where more appropriate. For #41184 Change-Id: I7eb85a498ed9788b42a636e775f261d755504ffa Reviewed-on: https://go-review.googlesource.com/c/go/+/361480 Trust: Tobias Klauser <tobias.klauser@gmail.com> Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2021-10-28all: go fix -fix=buildtag std cmd (except for bootstrap deps, vendor)Russ Cox
When these packages are released as part of Go 1.18, Go 1.16 will no longer be supported, so we can remove the +build tags in these files. Ran go fix -fix=buildtag std cmd and then reverted the bootstrapDirs as defined in src/cmd/dist/buildtool.go, which need to continue to build with Go 1.4 for now. Also reverted src/vendor and src/cmd/vendor, which will need to be updated in their own repos first. Manual changes in runtime/pprof/mprof_test.go to adjust line numbers. For #41184. Change-Id: Ic0f93f7091295b6abc76ed5cd6e6746e1280861e Reviewed-on: https://go-review.googlesource.com/c/go/+/344955 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2021-10-20runtime/internal/atomic: add atomic types for all functionsMichael Anthony Knyszek
Change-Id: I74f365316484feb819c31c77fbffd78fadfe32a9 Reviewed-on: https://go-review.googlesource.com/c/go/+/356169 Trust: Michael Knyszek <mknyszek@google.com> Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>
2021-06-30[dev.typeparams] all: merge master (4711bf3) into dev.typeparamsMatthew Dempsky
Conflicts: - src/cmd/compile/internal/walk/builtin.go On dev.typeparams, CL 330194 changed OCHECKNIL to not require manual SetTypecheck(1) anymore; while on master, CL 331070 got rid of the OCHECKNIL altogether by moving the check into the runtime support functions. - src/internal/buildcfg/exp.go On master, CL 331109 refactored the logic for parsing the GOEXPERIMENT string, so that it could be more easily reused by cmd/go; while on dev.typeparams, several CLs tweaked the regabi experiment defaults. Merge List: + 2021-06-30 4711bf30e5 doc/go1.17: linkify "language changes" in the runtime section + 2021-06-30 ed56ea73e8 path/filepath: deflake TestEvalSymlinksAboveRoot on darwin + 2021-06-30 c080d0323b cmd/dist: pass -Wno-unknown-warning-option in swig_callback_lto + 2021-06-30 7d0e9e6e74 image/gif: fix typo in the comment (io.ReadByte -> io.ByteReader) + 2021-06-30 0fa3265fe1 os: change example to avoid deprecated function + 2021-06-30 d19a53338f image: add Uniform.RGBA64At and Rectangle.RGBA64At + 2021-06-30 c45e800e0c crypto/x509: don't fail on optional auth key id fields + 2021-06-29 f9d50953b9 net: fix failure of TestCVE202133195 + 2021-06-29 e294b8a49e doc/go1.17: fix typo "MacOS" -> "macOS" + 2021-06-29 3463852b76 math/big: fix typo of comment (`BytesScanner` to `ByteScanner`) + 2021-06-29 fd4b587da3 cmd/compile: suppress details error for invalid variadic argument type + 2021-06-29 e2e05af6e1 cmd/internal/obj/arm64: fix an encoding error of CMPW instruction + 2021-06-28 4bb0847b08 cmd/compile,runtime: change unsafe.Slice((*T)(nil), 0) to return []T(nil) + 2021-06-28 1519271a93 spec: change unsafe.Slice((*T)(nil), 0) to return []T(nil) + 2021-06-28 5385e2386b runtime/internal/atomic: drop Cas64 pointer indirection in comments + 2021-06-28 956c81bfe6 cmd/go: add GOEXPERIMENT to `go env` output + 2021-06-28 a1d27269d6 cmd/go: prep for 'go env' refactoring + 2021-06-28 901510ed4e cmd/link/internal/ld: skip the windows ASLR test when CGO_ENABLED=0 + 2021-06-28 361159c055 cmd/cgo: fix 'see gmp.go' to 'see doc.go' + 2021-06-27 c95464f0ea internal/buildcfg: refactor GOEXPERIMENT parsing code somewhat + 2021-06-25 ed01ceaf48 runtime/race: use race build tag on syso_test.go + 2021-06-25 d1916e5e84 go/types: in TestCheck/issues.src, import regexp/syntax instead of cmd/compile/internal/syntax + 2021-06-25 5160896c69 go/types: in TestStdlib, import from source instead of export data + 2021-06-25 d01bc571f7 runtime: make ncgocall a global counter Change-Id: I1ce4a3b3ff7c824c67ad66dd27d9d5f1d25c0023
2021-06-28runtime/internal/atomic: drop Cas64 pointer indirection in commentsMia Zhu
Change-Id: Ieff0065cbd81e045594ce12e10338b0666816d70 GitHub-Last-Rev: d842f5cb3e5d75f87957c068f6accc9d4a4ac224 GitHub-Pull-Request: golang/go#46949 Reviewed-on: https://go-review.googlesource.com/c/go/+/331309 Trust: Keith Randall <khr@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-06-25[dev.typeparams] all: merge master (37f9a8f) into dev.typeparamsCuong Manh Le
Conflicts: - src/go/types/check_test.go CL 330629 fixed a bug in package qualification logic - src/internal/buildcfg/exp.go CL 329930 make parseExperiments get go arch string as input param Merge List: + 2021-06-25 37f9a8f69d go/types: fix a bug in package qualification logic + 2021-06-24 c309c89db5 reflect: document that InterfaceData is a low-entropy RNG + 2021-06-24 cce621431a cmd/compile: fix wrong type in SSA generation for OSLICE2ARRPTR + 2021-06-24 600a2a4ffb cmd/go: don't try to add replaced versions that won't be selected + 2021-06-24 a9bb38222a net: remove hard-coded timeout in dialClosedPort test helper + 2021-06-24 86d72fa2cb time: handle invalid UTF-8 byte sequences in quote to prevent panic + 2021-06-24 44a12e5f33 cmd/go: search breadth-first instead of depth-first for test dependency cycles + 2021-06-24 73496e0df0 net: use absDomainName in the Windows lookupPTR test helper + 2021-06-24 222ed1b38a os: enable TestFifoEOF on openbsd + 2021-06-22 0ebd5a8de0 cmd/go: update ToolTags based on GOARCH value + 2021-06-22 5bd09e5efc spec: unsafe.Add/Slice are not permitted in statement context + 2021-06-22 666315b4d3 runtime/internal/atomic: remove incorrect pointer indirection in comment + 2021-06-22 63daa774b5 go/types: guard against checking instantiation when generics is disabled + 2021-06-22 197a5ee2ab cmd/gofmt: remove stale documentation for the -G flag + 2021-06-22 9afd158eb2 go/parser: parse an ast.IndexExpr for a[] + 2021-06-21 1bd5a20e3c cmd/go: add a -go flag to 'go mod graph' + 2021-06-21 761edf71f6 cmd/internal/moddeps: use a temporary directory for GOMODCACHE if needed + 2021-06-21 a0400420ad cmd/internal/moddeps: use -mod=readonly instead of -mod=mod + 2021-06-21 3f9ec83b10 cmd/go: document GOPPC64 environment variable + 2021-06-21 20bdfba325 go/scanner: fall back to next() when encountering 0 bytes in parseIdentifier + 2021-06-21 44f9a3566c database/sql: fix deadlock test in prepare statement Change-Id: I16490e8ea70ee65081f467223857033842da513a
2021-06-22runtime/internal/atomic: remove incorrect pointer indirection in commentXing Gao
Change-Id: I9d743b7f6b001158299bea4af4aede678654bc8e GitHub-Last-Rev: 7e07834abc861e21030fe4a8eb323bac01e18f7a GitHub-Pull-Request: golang/go#46851 Reviewed-on: https://go-review.googlesource.com/c/go/+/329730 Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Ben Shi <powerman1st@163.com>
2021-06-17[dev.typeparams] runtime/internal/sys: replace BigEndian with ↵Michael Anthony Knyszek
goarch.BigEndian [generated] [git-generate] cd src/runtime/internal/atomic gofmt -w -r "sys.BigEndian -> goarch.BigEndian" . goimports -w *.go cd ../.. gofmt -w -r "sys.BigEndian -> goarch.BigEndian" . goimports -w *.go Change-Id: Iad35d2b367d8defb081a77ca837e7a7c805c2b7b Reviewed-on: https://go-review.googlesource.com/c/go/+/329190 Trust: Michael Knyszek <mknyszek@google.com> Run-TryBot: Michael Knyszek <mknyszek@google.com> Reviewed-by: Matthew Dempsky <mdempsky@google.com> TryBot-Result: Go Bot <gobot@golang.org>
2021-06-02[dev.typeparams] cmd/compile: add dictionary argument to generic functionsKeith Randall
When converting from a generic function to a concrete implementation, add a dictionary argument to the generic function (both an actual argument at each callsite, and a formal argument of each implementation). The dictionary argument comes before all other arguments (including any receiver). The dictionary argument is checked for validity, but is otherwise unused. Subsequent CLs will start using the dictionary for, e.g., converting a value of generic type to interface{}. Import/export required adding support for LINKSYMOFFSET, which is used by the dictionary checking code. Change-Id: I16a7a8d23c7bd6a897e0da87c69f273be9103bd7 Reviewed-on: https://go-review.googlesource.com/c/go/+/323272 Trust: Keith Randall <khr@golang.org> Trust: Dan Scales <danscales@google.com> Run-TryBot: Keith Randall <khr@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Dan Scales <danscales@google.com>
2021-06-02runtime: fix formattingKeith Randall
Fix up a gofmt complaint from CL 310591. Change-Id: I73534ef064a4cfc53539e5e65a8653e2cd684c64 Reviewed-on: https://go-review.googlesource.com/c/go/+/324090 Trust: Keith Randall <khr@golang.org> Run-TryBot: Keith Randall <khr@golang.org> Reviewed-by: Cherry Mui <cherryyz@google.com> TryBot-Result: Go Bot <gobot@golang.org>
2021-05-13all: add //go:build lines to assembly filesTobias Klauser
Don't add them to files in vendor and cmd/vendor though. These will be pulled in by updating the respective dependencies. For #41184 Change-Id: Icc57458c9b3033c347124323f33084c85b224c70 Reviewed-on: https://go-review.googlesource.com/c/go/+/319389 Trust: Tobias Klauser <tobias.klauser@gmail.com> Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Russ Cox <rsc@golang.org>
2021-05-12runtime/internal/atomic: add LSE atomics instructions to arm64Jonathan Swinney
As a follow up to an earlier change[1] to add ARMv8+LSE instructions in the compiler generated atomic intrinsics, make the same change in the runtime library. Since not all ARMv8 systems support LSE instructions, they are protected by a feature-flag branch. [1]: golang.org/cl/234217 commit: ecc3f5112eba Change-Id: I0e2fb22e78d5eddb6547863667a8865946679a00 Reviewed-on: https://go-review.googlesource.com/c/go/+/310591 Reviewed-by: Cherry Mui <cherryyz@google.com> Run-TryBot: Cherry Mui <cherryyz@google.com> TryBot-Result: Go Bot <gobot@golang.org> Trust: Heschi Kreinick <heschi@google.com>
2021-04-05runtime: extend internal atomics to comply with sync/atomicRuslan Andreev
The CV add changes according to TODO in Go source-code. Internal atomic set does not comply with sync/atomic library and has shortage operations for signed integers. This patch extend internal atomic set by Int32 and Int64 operations. It's implemented new aliases and asm versions of operations. As a result Cas64 was replaced by Casint64 in findRunnableGCWorker without type casting. Another purpose is unified structure of internal atomics' source code. Before, assembly impementations for different archs were in different files. For example, filename for AMD64 was asm_amd64.s, but filename for RISC-V was atomic_riscv64.s. Some arches have both files without any meaning. So, assembly files were merged and renamed to atomic_{$ARCH}.s filenames. Change-Id: I29a05a7cbf5f4a9cc146e8315536c038af545677 Reviewed-on: https://go-review.googlesource.com/c/go/+/289152 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Run-TryBot: Keith Randall <khr@golang.org> TryBot-Result: Go Bot <gobot@golang.org>
2021-02-20all: go fmt std cmd (but revert vendor)Russ Cox
Make all our package sources use Go 1.17 gofmt format (adding //go:build lines). Part of //go:build change (#41184). See https://golang.org/design/draft-gobuild Change-Id: Ia0534360e4957e58cd9a18429c39d0e32a6addb4 Reviewed-on: https://go-review.googlesource.com/c/go/+/294430 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-11-05cmd/compile: improve atomic swap intrinsics on arm64Jonathan Swinney
ARMv8.1 has added new instructions for atomic memory operations. This change builds on the previous change which added support for atomic add, 0a7ac93c27c9ade79fe0f66ae0bb81484c241ae5, to include similar support for atomic-compare-and-swap, atomic-swap, atomic-or, and atomic-and intrinsics. Since the new instructions are not guaranteed to be present, we guard their usages with a branch on a CPU feature. Peformance on an ARMv8.1 machine: name old time/op new time/op delta CompareAndSwap-16 37.9ns ±16% 24.1ns ± 4% -36.44% (p=0.000 n=10+9) CompareAndSwap64-16 38.6ns ±15% 24.1ns ± 3% -37.47% (p=0.000 n=10+10) name old time/op new time/op delta Swap-16 46.9ns ±32% 12.5ns ± 6% -73.40% (p=0.000 n=10+10) Swap64-16 53.4ns ± 1% 12.5ns ± 6% -76.56% (p=0.000 n=10+10) name old time/op new time/op delta Or8-16 8.81ns ± 0% 5.61ns ± 0% -36.32% (p=0.000 n=10+10) Or-16 7.21ns ± 0% 5.61ns ± 0% -22.19% (p=0.000 n=10+10) Or8Parallel-16 59.8ns ± 3% 12.5ns ± 2% -79.10% (p=0.000 n=10+10) OrParallel-16 51.7ns ± 3% 12.5ns ± 2% -75.84% (p=0.000 n=10+10) name old time/op new time/op delta And8-16 8.81ns ± 0% 5.61ns ± 0% -36.32% (p=0.000 n=10+10) And-16 7.21ns ± 0% 5.61ns ± 0% -22.19% (p=0.000 n=10+10) And8Parallel-16 59.1ns ± 6% 12.8ns ± 3% -78.33% (p=0.000 n=10+10) AndParallel-16 51.4ns ± 7% 12.8ns ± 3% -75.03% (p=0.000 n=10+10) Performance on an ARMv8.0 machine (no atomics instructions): name old time/op new time/op delta CompareAndSwap-16 61.3ns ± 0% 62.4ns ± 0% +1.70% (p=0.000 n=8+9) CompareAndSwap64-16 62.0ns ± 3% 61.3ns ± 2% ~ (p=0.093 n=10+10) name old time/op new time/op delta Swap-16 127ns ± 2% 131ns ± 2% +2.91% (p=0.001 n=10+10) Swap64-16 128ns ± 1% 131ns ± 2% +2.43% (p=0.001 n=10+10) name old time/op new time/op delta Or8-16 14.9ns ± 0% 15.3ns ± 0% +2.68% (p=0.000 n=10+10) Or-16 11.8ns ± 0% 12.3ns ± 0% +4.24% (p=0.000 n=10+10) Or8Parallel-16 137ns ± 1% 144ns ± 1% +4.97% (p=0.000 n=10+10) OrParallel-16 128ns ± 1% 136ns ± 1% +6.34% (p=0.000 n=10+10) name old time/op new time/op delta And8-16 14.9ns ± 0% 15.3ns ± 0% +2.68% (p=0.000 n=10+10) And-16 11.8ns ± 0% 12.3ns ± 0% +4.24% (p=0.000 n=10+10) And8Parallel-16 134ns ± 2% 141ns ± 1% +5.29% (p=0.000 n=10+10) AndParallel-16 125ns ± 2% 134ns ± 1% +7.10% (p=0.000 n=10+10) Fixes #39304 Change-Id: Idaca68701d4751650be6b4bedca3d57f51571712 Reviewed-on: https://go-review.googlesource.com/c/go/+/234217 Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Trust: fannie zhang <Fannie.Zhang@arm.com>
2020-10-23runtime/internal/atomic: add 32-bit And/OrMichael Pratt
These will be used in a following CL to perform larger bit clear and bit set than And8/Or8. Change-Id: I60f7b1099e29b69eb64add77564faee862880a8d Reviewed-on: https://go-review.googlesource.com/c/go/+/260977 Run-TryBot: Michael Pratt <mpratt@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Trust: Michael Pratt <mpratt@google.com>
2020-10-21cmd/compiler,cmd/go,sync: add internal {LoadAcq,StoreRel}64 on ppc64Paul E. Murphy
Add an internal atomic intrinsic for load with acquire semantics (extending LoadAcq to 64b) and add LoadAcquintptr for internal use within the sync package. For other arches, this remaps to the appropriate atomic.Load{,64} intrinsic which should not alter code generation. Similarly, add StoreRel{uintptr,64} for consistency, and inline. Finally, add an exception to allow sync to directly use the runtime/internal/atomic package which avoids more convoluted workarounds (contributed by Lynn Boger). In an extreme example, sync.(*Pool).pin consumes 20% of wall time during fmt tests. This is reduced to 5% on ppc64le/power9. From the fmt benchmarks on ppc64le: name old time/op new time/op delta SprintfPadding 468ns ± 0% 451ns ± 0% -3.63% SprintfEmpty 73.3ns ± 0% 51.9ns ± 0% -29.20% SprintfString 135ns ± 0% 122ns ± 0% -9.63% SprintfTruncateString 232ns ± 0% 214ns ± 0% -7.76% SprintfTruncateBytes 216ns ± 0% 202ns ± 0% -6.48% SprintfSlowParsingPath 162ns ± 0% 142ns ± 0% -12.35% SprintfQuoteString 1.00µs ± 0% 0.99µs ± 0% -1.39% SprintfInt 117ns ± 0% 104ns ± 0% -11.11% SprintfIntInt 190ns ± 0% 175ns ± 0% -7.89% SprintfPrefixedInt 232ns ± 0% 212ns ± 0% -8.62% SprintfFloat 270ns ± 0% 255ns ± 0% -5.56% SprintfComplex 1.01µs ± 0% 0.99µs ± 0% -1.68% SprintfBoolean 127ns ± 0% 111ns ± 0% -12.60% SprintfHexString 220ns ± 0% 198ns ± 0% -10.00% SprintfHexBytes 261ns ± 0% 252ns ± 0% -3.45% SprintfBytes 600ns ± 0% 590ns ± 0% -1.67% SprintfStringer 684ns ± 0% 658ns ± 0% -3.80% SprintfStructure 2.57µs ± 0% 2.57µs ± 0% -0.12% ManyArgs 669ns ± 0% 646ns ± 0% -3.44% FprintInt 140ns ± 0% 136ns ± 0% -2.86% FprintfBytes 184ns ± 0% 181ns ± 0% -1.63% FprintIntNoAlloc 140ns ± 0% 136ns ± 0% -2.86% ScanInts 929µs ± 0% 921µs ± 0% -0.79% ScanRecursiveInt 122ms ± 0% 121ms ± 0% -0.11% ScanRecursiveIntReaderWrapper 122ms ± 0% 122ms ± 0% -0.18% Change-Id: I4d66780261b57b06ef600229e475462e7313f0d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/253748 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> Reviewed-by: Lynn Boger <laboger@linux.vnet.ibm.com> Reviewed-by: Keith Randall <khr@golang.org> Trust: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Go Bot <gobot@golang.org>
2020-10-16runtime/internal/atomic: panic nicely on unaligned 64-bit atomicsAustin Clements
On 386 and arm, unaligned 64-bit atomics aren't safe, so we check for this and panic. Currently, we panic by dereferencing nil, which may be expedient but is pretty user-hostile since it gives no hint of what the actual problem was. This CL replaces this with an actual panic. The only subtlety here is now the atomic assembly implementations are calling back into Go, so they have to play nicely with stack maps and stack scanning. On 386, this just requires declaring NO_LOCAL_POINTERS. On arm, this is somewhat more complicated: first, we have to move the alignment check into the functions that have Go signatures. Then we have to support both the tail call from these functions to the underlying implementation (which requires that they have no frame) and the call into Go to panic (which requires that they have a frame). We resolve this by forcing them to have no frame and setting up the frame manually just before the panic call. Change-Id: I19f1e860045df64088013db37a18acea47342c69 Reviewed-on: https://go-review.googlesource.com/c/go/+/262778 Trust: Austin Clements <austin@google.com> Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com>
2020-10-16runtime/internal/atomic: drop package prefixesAustin Clements
This drops package prefixes from the assembly code on 386 and arm. In addition to just being nicer, this allows the assembler to automatically pick up the argument stack map from the Go signatures of these functions. This doesn't matter right now because these functions never call back out to Go, but prepares us for the next CL. Change-Id: I90fed7d4dd63ad49274529c62804211b6390e2e9 Reviewed-on: https://go-review.googlesource.com/c/go/+/262777 Trust: Austin Clements <austin@google.com> Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com>
2020-08-24runtime: add test for StorepNoWB param leakingCuong Manh Le
CL 249962 added wasm StorepNoWB implementation in assembly, it's now like all other architectures. This CL adds a general test that the second param of StorepNoWB must be force to escape. Fixes #40975 Change-Id: I1eccc7e50a3ec742a1912d65f25b15f9f5ad9241 Reviewed-on: https://go-review.googlesource.com/c/go/+/249761 Run-TryBot: Cuong Manh Le <cuong.manhle.vn@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2020-08-23runtime: implement StorepNoWB for wasm in assemblyKeith Randall
The second argument of StorepNoWB must be forced to escape. The current Go code does not explicitly enforce that property. By implementing in assembly, and not using go:noescape, we force the issue. Test is in CL 249761. Issue #40975. This CL is needed for CL 249917, which changes how go:notinheap works and breaks the previous StorepNoWB wasm code. I checked for other possible errors like this. This is the only go:notinheap that isn't in the runtime itself. Change-Id: I43400a806662655727c4a3baa8902b63bdc9fa57 Reviewed-on: https://go-review.googlesource.com/c/go/+/249962 Run-TryBot: Keith Randall <khr@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2020-04-03runtime: eliminate redundant load in Xchg and Xadd on arm64Xiangdong Ji
Loading arguments of Xchg(64) and Xadd(64) functions to registers could be done only once. Change-Id: Iaf0a695ec9c6a221dfa755855edb68c476978a5b Reviewed-on: https://go-review.googlesource.com/c/go/+/227001 Reviewed-by: Keith Randall <khr@golang.org> Run-TryBot: Keith Randall <khr@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-03-15cmd/asm,cmd/internal/obj/riscv: add atomic memory operation instructionsJoel Sing
Use instructions in place of currently used defines. Updates #36765 Change-Id: I00bb59e77b1aace549d7857cc9721ba2cb4ac6ca Reviewed-on: https://go-review.googlesource.com/c/go/+/220541 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-03-15cmd/asm,cmd/internal/obj/riscv: add LR/SC instructionsJoel Sing
Add support for Load-Reserved (LR) and Store-Conditional (SC) instructions. Use instructions in place of currently used defines. Updates #36765 Change-Id: I77e660639802293ece40cfde4865ac237e3308d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/220540 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-03-15cmd/internal/obj/riscv: add FENCE instructionJoel Sing
Also remove #define's that were previously in use. Updates #36765 Change-Id: I90b6a8629c78f549012f3f6c5f3b325336182712 Reviewed-on: https://go-review.googlesource.com/c/go/+/220539 Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-01-19runtime: add support for linux/riscv64Joel Sing
Based on riscv-go port. Updates #27532 Change-Id: If522807a382130be3c8d40f4b4c1131d1de7c9e3 Reviewed-on: https://go-review.googlesource.com/c/go/+/204632 Run-TryBot: Joel Sing <joel@sing.id.au> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2019-11-11cmd/compile, runtime: intrinsify atomic And8 and Or8 on s390xMichael Munday
Intrinsify these functions to match other platforms. Update the sequence of instructions used in the assembly implementations to match the intrinsics. Also, add a micro benchmark so we can more easily measure the performance of these two functions: name old time/op new time/op delta And8-8 5.33ns ± 7% 2.55ns ± 8% -52.12% (p=0.000 n=20+20) And8Parallel-8 7.39ns ± 5% 3.74ns ± 4% -49.34% (p=0.000 n=20+20) Or8-8 4.84ns ±15% 2.64ns ±11% -45.50% (p=0.000 n=20+20) Or8Parallel-8 7.27ns ± 3% 3.84ns ± 4% -47.10% (p=0.000 n=19+20) By using a 'rotate then xor selected bits' instruction combined with either a 'load and and' or a 'load and or' instruction we can implement And8 and Or8 with far fewer instructions. Replacing 'compare and swap' with atomic instructions may also improve performance when there is contention. Change-Id: I28bb8032052b73ae8ccdf6e4c612d2877085fa01 Reviewed-on: https://go-review.googlesource.com/c/go/+/204277 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-11-01[dev.link] all: merge branch 'master' into dev.linkThan McIntosh
Fixed a couple of minor conflicts in lib.go and deadcode.go relating to debug logging. Change-Id: I58335fc42ab1f1f3409fd8354da4f26419e8fb22
2019-10-31runtime/internal/atomic: add tests for And8 and Or8Michael Munday
Add some simple unit tests for these atomic operations. These can't catch all the bugs that are possible with these operations but at least they provide some coverage. Change-Id: I94b9f451fcc9fecdb2a1448c5357b019563ad275 Reviewed-on: https://go-review.googlesource.com/c/go/+/204317 Run-TryBot: Michael Munday <mike.munday@ibm.com> Reviewed-by: Austin Clements <austin@google.com>
2019-10-29runtime/internal/atomic: add Store8Austin Clements
We already have Load8, And8, and Or8. For #10958, #24543, but makes sense on its own. Change-Id: I478529fc643edc57efdeccaae413c99edd19b2eb Reviewed-on: https://go-review.googlesource.com/c/go/+/203283 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-10-21[dev.link] cmd/link: do not put static symbols into name lookup tableCherry Zhang
Since the previous CL, we will not reference static symbols by name. Therefore no need to put them into the name lookup table. On Linux/ARM, in runtime/internal/atomic/sys_linux_arm.s, the kernelcas function has a definition and a reference written in two different forms, one with package prefix, one without. This way, the assembler cannot know they are the same symbol, only the linker knows. This is quite unusual, unify the names to so the assembler can resolve it to index. Change-Id: Ie7223097be6a3b65f3fa43ed4575da9972ef5b69 Reviewed-on: https://go-review.googlesource.com/c/go/+/201998 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Than McIntosh <thanm@google.com>
2019-10-10all: remove nacl (part 3, more amd64p32)Brad Fitzpatrick
Part 1: CL 199499 (GOOS nacl) Part 2: CL 200077 (amd64p32 files, toolchain) Part 3: stuff that arguably should've been part of Part 2, but I forgot one of my grep patterns when splitting the original CL up into two parts. This one might also have interesting stuff to resurrect for any future x32 ABI support. Updates #30439 Change-Id: I2b4143374a253a003666f3c69e776b7e456bdb9c Reviewed-on: https://go-review.googlesource.com/c/go/+/200318 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-10-09all: remove the nacl port (part 2, amd64p32 + toolchain)Brad Fitzpatrick
This is part two if the nacl removal. Part 1 was CL 199499. This CL removes amd64p32 support, which might be useful in the future if we implement the x32 ABI. It also removes the nacl bits in the toolchain, and some remaining nacl bits. Updates #30439 Change-Id: I2475d5bb066d1b474e00e40d95b520e7c2e286e1 Reviewed-on: https://go-review.googlesource.com/c/go/+/200077 Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-06-06runtime/internal/atomic: remove erroneous ABI wrappersAustin Clements
CL 179862 introduced go:linkname directives to create ABI wrappers for Store and Store64 on s390x, but a concurrent change (CL 180439) replaced the Go definitions of these functions with assembly definitions. This resulted in conflicting definitions for the ABI0 symbols, which led to a bootstrap linking failure. Fix this by removing the now-incorrect go:linkname directives for Store and Store64. This should fix the linux-s390x builders. Updates #31230. Change-Id: I8de8c03c23412fc217d428c0018cc56eb2f9996f Reviewed-on: https://go-review.googlesource.com/c/go/+/181078 Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-06-06runtime/internal/atomic: export more ABI0 wrappersAustin Clements
Somehow I missed these two functions in CL 179863. This should fix the linux-arm builders. Updates #31230. Change-Id: I3f8bef3fac331b505a55c0850b0fbc799b7c06c5 Reviewed-on: https://go-review.googlesource.com/c/go/+/181077 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Bryan C. Mills <bcmills@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-06-06runtime: mark all Go symbols called from assembly in other packagesAustin Clements
This marks all Go symbols called from assembly in other packages with "go:linkname" directives to ensure they get ABI wrappers. Now that we have this go:linkname convention, this also removes the abi0Syms definition in the runtime, which was used to give morestackc an ABI0 wrapper. Instead, we now just mark morestackc with a go:linkname directive. This was tested with buildall.bash in the default configuration, with -race, and with -gcflags=all=-d=ssa/intrinsics/off. Since I couldn't test cgo on non-Linux configurations, I manually grepped for runtime symbols in runtime/cgo. Updates #31230. Change-Id: I6c8aa56be2ca6802dfa2bf159e49c411b9071bf1 Reviewed-on: https://go-review.googlesource.com/c/go/+/179862 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>
2019-06-06cmd/compile, runtime: make atomic loads/stores sequentially consistent on s390xMichael Munday
The z/Architecture does not guarantee that a load following a store will not be reordered with that store, unless they access the same address. Therefore if we want to ensure the sequential consistency of atomic loads and stores we need to perform serialization operations after atomic stores. We do not need to serialize in the runtime when using StoreRel[ease] and LoadAcq[uire]. The z/Architecture already provides sufficient ordering guarantees for these operations. name old time/op new time/op delta AtomicLoad64-16 0.51ns ± 0% 0.51ns ± 0% ~ (all equal) AtomicStore64-16 0.51ns ± 0% 0.60ns ± 9% +16.47% (p=0.000 n=17+20) AtomicLoad-16 0.51ns ± 0% 0.51ns ± 0% ~ (all equal) AtomicStore-16 0.51ns ± 0% 0.60ns ± 9% +16.50% (p=0.000 n=18+20) Fixes #32428. Change-Id: I88d19a4010c46070e4fff4b41587efe4c628d4d9 Reviewed-on: https://go-review.googlesource.com/c/go/+/180439 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 18:18:50 +0000