go/src/runtime/mgcsweep.go, branch fix-runtime-test-GOMAXPROCS

runtime: have mergeInlineMarkBits also clear the inline mark bits

2025-07-15T19:24:28Z

This is conceptually simpler, as the sweeper doesn't have to worry about clearing them separately. It also doesn't have a use for them. This will also be useful to avoiding unnecessary zeroing in initInlineMarkBits at allocation time. Currently, because it's used in both span allocation and at sweep time, we cannot blindly trust needzero. This change also renames mergeInlineMarkBits to moveInlineMarkBits to make this change in semantics clearer from the name. For #73581. Change-Id: Ib154738a945633b7ff5b2ae27235baa310400139 Reviewed-on: https://go-review.googlesource.com/c/go/+/687936 Auto-Submit: Michael Knyszek LUCI-TryBot-Result: Go LUCI Reviewed-by: Michael Pratt

runtime: add valgrind instrumentation

2025-05-21T17:08:08Z

Add build tag gated Valgrind annotations to the runtime which let it understand how the runtime manages memory. This allows for Go binaries to be run under Valgrind without emitting spurious errors. Instead of adding the Valgrind headers to the tree, and using cgo to call the various Valgrind client request macros, we just add an assembly function which emits the necessary instructions to trigger client requests. In particular we add instrumentation of the memory allocator, using a two-level mempool structure (as described in the Valgrind manual [0]). We also add annotations which allow Valgrind to track which memory we use for stacks, which seems necessary to let it properly function. We describe the memory model to Valgrind as follows: we treat heap arenas as a "pool" created with VALGRIND_CREATE_MEMPOOL_EXT (so that we can use VALGRIND_MEMPOOL_METAPOOL and VALGRIND_MEMPOOL_AUTO_FREE). Within the pool we treat spans as "superblocks", annotated with VALGRIND_MEMPOOL_ALLOC. We then allocate individual objects within spans with VALGRIND_MALLOCLIKE_BLOCK. It should be noted that running binaries under Valgrind can be _quite slow_, and certain operations, such as running the GC, can be _very slow_. It is recommended to run programs with GOGC=off. Additionally, async preemption should be turned off, since it'll cause strange behavior (GODEBUG=asyncpreemptoff=1). Running Valgrind with --leak-check=yes will result in some errors resulting from some things not being marked fully free'd. These likely need more annotations to rectify, but for now it is recommended to run with --leak-check=off. Updates #73602 [0] https://valgrind.org/docs/manual/mc-manual.html#mc-manual.mempools Change-Id: I71b26c47d7084de71ef1e03947ef6b1cc6d38301 Reviewed-on: https://go-review.googlesource.com/c/go/+/674077 LUCI-TryBot-Result: Go LUCI Reviewed-by: Michael Knyszek

runtime: add goschedIfBusy to bgsweep to prevent livelock after inlining

2025-05-12T17:07:14Z

gcMarkTermination() ensures that all caches are flushed before continuing the GC cycle, thus preempting all goroutines. However, inlining calls to lock() in bgsweep makes it non-preemptible for most of the time, leading to livelock. This change adds explicit preemption to avoid this. Fixes #73499. Change-Id: I4abf0d658f3d7a03ad588469cd013a0639de0c8a Reviewed-on: https://go-review.googlesource.com/c/go/+/668795 LUCI-TryBot-Result: Go LUCI Auto-Submit: Michael Knyszek Reviewed-by: Michael Knyszek Reviewed-by: Michael Pratt

runtime: schedule cleanups across multiple goroutines

2025-05-08T18:10:33Z

This change splits the finalizer and cleanup queues and implements a new lock-free blocking queue for cleanups. The basic design is as follows: The cleanup queue is organized in fixed-sized blocks. Individual cleanup functions are queued, but only whole blocks are dequeued. Enqueuing cleanups places them in P-local cleanup blocks. These are flushed to the full list as they get full. Cleanups can only be enqueued by an active sweeper. Dequeuing cleanups always dequeues entire blocks from the full list. Cleanup blocks can be dequeued and executed at any time. The very last active sweeper in the sweep phase is responsible for flushing all local cleanup blocks to the full list. It can do this without any synchronization because the next GC can't start yet, so we can be very certain that nobody else will be accessing the local blocks. Cleanup blocks are stored off-heap because the need to be allocated by the sweeper, which is called from heap allocation paths. As a result, the GC treats cleanup blocks as roots, just like finalizer blocks. Flushes to the full list signal to the scheduler that cleanup goroutines should be awoken. Every time the scheduler goes to wake up a cleanup goroutine and there were more signals than goroutines to wake, it then forwards this signal to runtime.AddCleanup, so that it creates another goroutine the next time it is called, up to gomaxprocs goroutines. The signals here are a little convoluted, but exist because the sweeper and the scheduler cannot safely create new goroutines. For #71772. For #71825. Change-Id: Ie839fde2b67e1b79ac1426be0ea29a8d923a62cc Reviewed-on: https://go-review.googlesource.com/c/go/+/650697 Reviewed-by: Michael Pratt LUCI-TryBot-Result: Go LUCI Auto-Submit: Michael Knyszek

runtime: fix condition to emit gcpacertrace end-of-sweep line

2025-05-08T14:50:10Z

It's the job of the last sweeper to emit the GC pacer trace. The last sweeper can identify themselves by reducing the count of sweepers, and also seeing that there's no more sweep work. Currently this identification is broken, however, because the last sweeper doesn't check the state they just transitioned sweeping into, but rather the state they transitioned from (one sweeper, no sweep work left). By design, it's impossible to transition *out* of this state, except for another GC to start, but that doesn't take this codepath. This means lines like pacer: sweep done at heap size ... were missing from the gcpacertrace output for a long time. This change fixes this problem by having the last sweeper check the state they just transitioned sweeping to, instead of the state they transitioned from. Change-Id: I44bcd32fe2c8ae6ac6c21ba6feb2e7b9e17f60cc Reviewed-on: https://go-review.googlesource.com/c/go/+/670735 LUCI-TryBot-Result: Go LUCI Reviewed-by: Michael Pratt

runtime: mark and scan small objects in whole spans [green tea]

2025-05-02T17:28:07Z

Our current parallel mark algorithm suffers from frequent stalls on memory since its access pattern is essentially random. Small objects are the worst offenders, since each one forces pulling in at least one full cache line to access even when the amount to be scanned is far smaller than that. Each object also requires an independent access to per-object metadata. The purpose of this change is to improve garbage collector performance by scanning small objects in batches to obtain better cache locality than our current approach. The core idea behind this change is to defer marking and scanning small objects, and then scan them in batches localized to a span. This change adds scanned bits to each small object (<=512 bytes) span in addition to mark bits. The scanned bits indicate that the object has been scanned. (One way to think of them is "grey" bits and "black" bits in the tri-color mark-sweep abstraction.) Each of these spans is always 8 KiB and if they contain pointers, the pointer/scalar data is already packed together at the end of the span, allowing us to further optimize the mark algorithm for this specific case. When the GC encounters a pointer, it first checks if it points into a small object span. If so, it is first marked in the mark bits, and then the object is queued on a work-stealing P-local queue. This object represents the whole span, and we ensure that a span can only appear at most once in any queue by maintaining an atomic ownership bit for each span. Later, when the pointer is dequeued, we scan every object with a set mark that doesn't have a corresponding scanned bit. If it turns out that was the only object in the mark bits since the last time we scanned the span, we scan just that object directly, essentially falling back to the existing algorithm. noscan objects have no scan work, so they are never queued. Each span's mark and scanned bits are co-located together at the end of the span. Since the span is always 8 KiB in size, it can be found with simple pointer arithmetic. Next to the marks and scans we also store the size class, eliminating the need to access the span's mspan altogether. The work-stealing P-local queue is a new source of GC work. If this queue gets full, half of it is dumped to a global linked list of spans to scan. The regular scan queues are always prioritized over this queue to allow time for darts to accumulate. Stealing work from other Ps is a last resort. This change also adds a new debug mode under GODEBUG=gctrace=2 that dumps whole-span scanning statistics by size class on every GC cycle. A future extension to this CL is to use SIMD-accelerated scanning kernels for scanning spans with high mark bit density. For #19112. (Deadlock averted in GOEXPERIMENT.) For #73581. Change-Id: I4bbb4e36f376950a53e61aaaae157ce842c341bc Reviewed-on: https://go-review.googlesource.com/c/go/+/658036 Auto-Submit: Michael Knyszek Reviewed-by: Michael Pratt LUCI-TryBot-Result: Go LUCI

runtime: move sizeclass defs to new package internal/runtime/gc

2025-04-23T15:00:33Z

We will want to reference these definitions from new generator programs, and this is a good opportunity to cleanup all these old C-style names. Change-Id: Ifb06f0afc381e2697e7877f038eca786610c96de Reviewed-on: https://go-review.googlesource.com/c/go/+/655275 Auto-Submit: Michael Knyszek LUCI-TryBot-Result: Go LUCI Reviewed-by: Cherry Mui Reviewed-by: Michael Pratt

weak: move internal/weak to weak, and update according to proposal

2024-11-18T22:29:23Z

The updates are: - API documentation changes. - Removal of the old package documentation discouraging linkname. - Addition of new package documentation with some advice. - Renaming of weak.Pointer.Strong -> weak.Pointer.Value. Fixes #67552. Change-Id: Ifad7e629b6d339dacaf2ca37b459d7f903e31bf8 Reviewed-on: https://go-review.googlesource.com/c/go/+/628455 LUCI-TryBot-Result: Go LUCI Reviewed-by: Carlos Amedee Auto-Submit: Michael Knyszek

runtime: get rid of gc programs for types

2024-11-18T20:36:14Z

Instead, have the runtime build the gc bitmaps on demand at runtime. Change-Id: If7a245bc62e4bce3ce80972410b0ed307d921abe Reviewed-on: https://go-review.googlesource.com/c/go/+/616255 LUCI-TryBot-Result: Go LUCI Reviewed-by: Cherry Mui Reviewed-by: Keith Randall

runtime: add race detector tips to reportZombies func

2024-11-15T17:47:27Z

We can find a few issues finally turned out to be a race condition, such as #47513. I believe such a tip can eliminate the need for developers to file this kind of issue in the first place. Change-Id: I1597fa09fde641882e8e87453470941747705272 GitHub-Last-Rev: 9f136f5b3bee78f90f434dcea1cabf397c6c05f2 GitHub-Pull-Request: golang/go#70331 Reviewed-on: https://go-review.googlesource.com/c/go/+/627816 LUCI-TryBot-Result: Go LUCI Auto-Submit: Michael Pratt Reviewed-by: Cherry Mui Reviewed-by: Jorropo Reviewed-by: Michael Pratt