go/src/runtime/mpagecache.go, branch makepkg

runtime,internal: move runtime/internal/sys to internal/runtime/sys

2024-07-23T19:05:35Z

Cleanup and friction reduction For #65355. Change-Id: Ia14c9dc584a529a35b97801dd3e95b9acc99a511 Reviewed-on: https://go-review.googlesource.com/c/go/+/600436 Reviewed-by: Keith Randall LUCI-TryBot-Result: Go LUCI Reviewed-by: Keith Randall

runtime: manage huge pages explicitly

2023-04-19T14:30:00Z

This change makes it so that on Linux the Go runtime explicitly marks page heap memory as either available to be backed by hugepages or not using heuristics based on density. The motivation behind this change is twofold: 1. In default Linux configurations, khugepaged can recoalesce hugepages even after the scavenger breaks them up, resulting in significant overheads for small heaps when their heaps shrink. 2. The Go runtime already has some heuristics about this, but those heuristics appear to have bit-rotted and result in haphazard hugepage management. Unlucky (but otherwise fairly dense) regions of memory end up not backed by huge pages while sparse regions end up accidentally marked MADV_HUGEPAGE and are not later broken up by the scavenger, because it already got the memory it needed from more dense sections (this is more likely to happen with small heaps that go idle). In this change, the runtime uses a new policy: 1. Mark all new memory MADV_HUGEPAGE. 2. Track whether each page chunk (4 MiB) became dense during the GC cycle. Mark those MADV_HUGEPAGE, and hide them from the scavenger. 3. If a chunk is not dense for 1 full GC cycle, make it visible to the scavenger. 4. The scavenger marks a chunk MADV_NOHUGEPAGE before it scavenges it. This policy is intended to try and back memory that is a good candidate for huge pages (high occupancy) with huge pages, and give memory that is not (low occupancy) to the scavenger. Occupancy is defined not just by occupancy at any instant of time, but also occupancy in the near future. It's generally true that by the end of a GC cycle the heap gets quite dense (from the perspective of the page allocator). Because we want scavenging and huge page management to happen together (the right time to MADV_NOHUGEPAGE is just before scavenging in order to break up huge pages and keep them that way) and the cost of applying MADV_HUGEPAGE and MADV_NOHUGEPAGE is somewhat high, the scavenger avoids releasing memory in dense page chunks. All this together means the scavenger will now more generally release memory on a ~1 GC cycle delay. Notably this has implications for scavenging to maintain the memory limit and the runtime/debug.FreeOSMemory API. This change makes it so that in these cases all memory is visible to the scavenger regardless of sparseness and delays the page allocator in re-marking this memory with MADV_NOHUGEPAGE for around 1 GC cycle to mitigate churn. The end result of this change should be little-to-no performance difference for dense heaps (MADV_HUGEPAGE works a lot like the default unmarked state) but should allow the scavenger to more effectively take back fragments of huge pages. The main risk here is churn, because MADV_HUGEPAGE usually forces the kernel to immediately back memory with a huge page. That's the reason for the large amount of hysteresis (1 full GC cycle) and why the definition of high density is 96% occupancy. Fixes #55328. Change-Id: I8da7998f1a31b498a9cc9bc662c1ae1a6bf64630 Reviewed-on: https://go-review.googlesource.com/c/go/+/436395 Reviewed-by: Michael Pratt Run-TryBot: Michael Knyszek TryBot-Result: Gopher Robot

runtime: fix incorrect comment

2023-01-11T22:26:01Z

Fix incorrect comment for the runtime package. Change-Id: Iab889eff0e9c622afbed959d32b8b5f0ed0bfebf GitHub-Last-Rev: e9587868db36d9dd466f52f7ffc5de09947a6b2f GitHub-Pull-Request: golang/go#57731 Reviewed-on: https://go-review.googlesource.com/c/go/+/461498 Reviewed-by: Keith Randall TryBot-Result: Gopher Robot Run-TryBot: Keith Randall Auto-Submit: Keith Randall Reviewed-by: Ian Lance Taylor Reviewed-by: Keith Randall

runtime: change maxSearchAddr into a helper function

2022-05-11T03:28:01Z

This avoids a dependency on the compiler statically initializing maxSearchAddr, which is necessary so we can disable the (overly aggressive and spec non-conforming) optimizations in cmd/compile and gccgo. Updates #51913. Change-Id: I424e62c81c722bb179ed8d2d8e188274a1aeb7b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/396194 Reviewed-by: Michael Knyszek Reviewed-by: Austin Clements Run-TryBot: Matthew Dempsky TryBot-Result: Gopher Robot

runtime: set and clear only the relevant bits in allocToCache

2021-11-04T20:01:22Z

Currently allocToCache ham-handedly calls pageAlloc.allocRange on the full size of the cache. This is fine as long as scavenged bits are never set when alloc bits are set. This is true right now, but won't be true as of the next CL. This change makes allocToCache more carefully set the bits. Note that in the allocToCache path, we were also calling update *twice*, erroneously. The first time, with contig=true! Luckily today there's no correctness error there because the page cache is small enough that the contig=true logic doesn't matter, but this should at least improve allocation performance a little bit. Change-Id: I3ff9590ac86d251e4c5063cfd633570238b0cdbf Reviewed-on: https://go-review.googlesource.com/c/go/+/356609 Trust: Michael Knyszek Run-TryBot: Michael Knyszek TryBot-Result: Go Bot Reviewed-by: Michael Pratt

runtime: add heap lock assertions

2020-10-30T20:21:14Z

Some functions that required holding the heap lock _or_ world stop have been simplified to simply requiring the heap lock. This is conceptually simpler and taking the heap lock during world stop is guaranteed to not contend. This was only done on functions already called on the systemstack to avoid too many extra systemstack calls in GC. Updates #40677 Change-Id: I15aa1dadcdd1a81aac3d2a9ecad6e7d0377befdc Reviewed-on: https://go-review.googlesource.com/c/go/+/250262 Run-TryBot: Michael Pratt TryBot-Result: Go Bot Reviewed-by: Austin Clements Trust: Michael Pratt

runtime: rename pageAlloc receiver

2020-10-23T16:54:35Z

The history of pageAlloc using 's' as a receiver are lost to the depths of time (perhaps it used to be called summary?), but it doesn't make much sense anymore. Rename it to 'p'. Generated with: $ cd src/runtime $ grep -R -b "func (s \*pageAlloc" . | awk -F : '{ print $1 ":#" $2+6 }' | xargs -n 1 -I {} env GOROOT=$(pwd)/../../ gorename -offset {} -to p -v $ grep -R -b "func (s \*pageAlloc" . | awk -F : '{ print $1 ":#" $2+6 }' | xargs -n 1 -I {} env GOROOT=$(pwd)/../../ GOARCH=386 gorename -offset {} -to p -v $ GOROOT=$(pwd)/../../ gorename -offset mpagecache.go:#2397 -to p -v ($2+6 to advance past "func (".) Plus manual comment fixups. Change-Id: I2d521a1cbf6ebe2ef6aae92e654bfc33c63d1aa9 Reviewed-on: https://go-review.googlesource.com/c/go/+/250517 Trust: Michael Pratt Run-TryBot: Michael Pratt TryBot-Result: Go Bot Reviewed-by: Michael Knyszek

runtime: use offAddr in more parts of the runtime

2020-05-08T16:32:03Z

This change uses the new offAddr type in more parts of the runtime where we've been implicitly switching from the default address space to a contiguous view. The purpose of offAddr is to represent addresses in the contiguous view of the address space, and to make direct computations between real addresses and offset addresses impossible. This change thus improves readability in the runtime. Updates #35788. Change-Id: I4e1c5fed3ed68aa12f49a42b82eb3f46aba82fc1 Reviewed-on: https://go-review.googlesource.com/c/go/+/230718 Run-TryBot: Michael Knyszek TryBot-Result: Gobot Gobot Reviewed-by: Austin Clements

runtime: ensure allocToCache updates searchAddr in a valid way

2020-04-27T21:37:31Z

Currently allocToCache assumes it can move the search address past the block it allocated the cache from, which violates the property that searchAddr should always point to mapped memory (i.e. memory represented by pageAlloc.inUse). This bug was already fixed once for pageAlloc.alloc in the Go 1.14 release via CL 216697, but that changed failed to take into account allocToCache. Fixes #38605. Change-Id: Id08180aa10d19dc0f9f551a1d9e327a295560dff Reviewed-on: https://go-review.googlesource.com/c/go/+/229577 Run-TryBot: Michael Knyszek TryBot-Result: Gobot Gobot Reviewed-by: David Chase

runtime: check the correct sanity condition in the page allocator

2020-03-30T14:30:38Z

Currently there are a few sanity checks in the page allocator which should fail immediately but because it's a check for a negative number on a uint, it's actually dead-code. If there's a bug in the page allocator which would cause the sanity check to fail, this could cause memory corruption by returning an invalid address (more precisely, one might either see a segfault, or span overlap). This change fixes these sanity checks to check the correct condition. Fixes #38130. Change-Id: Ia19786cece783d39f26df24dec8788833a6a3f21 Reviewed-on: https://go-review.googlesource.com/c/go/+/226297 Reviewed-by: Giovanni Bajo Reviewed-by: Cherry Zhang