aboutsummaryrefslogtreecommitdiff
path: root/src/runtime/malloc.go
AgeCommit message (Collapse)Author
2019-11-08runtime: remove old page allocatorMichael Anthony Knyszek
This change removes the old page allocator from the runtime. Updates #35112. Change-Id: Ib20e1c030f869b6318cd6f4288a9befdbae1b771 Reviewed-on: https://go-review.googlesource.com/c/go/+/195700 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-11-07runtime: switch to new page allocatorMichael Anthony Knyszek
This change flips the oldPageAllocator constant enabling the new page allocator in the Go runtime. Updates #35112. Change-Id: I7fc8332af9fd0e43ce28dd5ebc1c1ce519ce6d0c Reviewed-on: https://go-review.googlesource.com/c/go/+/201765 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-11-07runtime: integrate new page allocator into runtimeMichael Anthony Knyszek
This change integrates all the bits and pieces of the new page allocator into the runtime, behind a global constant. Updates #35112. Change-Id: I6696bde7bab098a498ab37ed2a2caad2a05d30ec Reviewed-on: https://go-review.googlesource.com/c/go/+/201764 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-11-07runtime: add scavenging code for new page allocatorMichael Anthony Knyszek
This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-11-07runtime: define darwin/arm64's address space as 33 bitsMichael Anthony Knyszek
On iOS, the address space is not 48 bits as one might believe, since it's arm64 hardware. In fact, all pointers are truncated to 33 bits, and the OS only gives applications access to the range [1<<32, 2<<32). While today this has no effect on the Go runtime, future changes which care about address space size need this to be correct. Updates #35112. Change-Id: Id518a2298080f7e3d31cf7d909506a37748cc49a Reviewed-on: https://go-review.googlesource.com/c/go/+/205758 Run-TryBot: Michael Knyszek <mknyszek@google.com> Reviewed-by: Keith Randall <khr@golang.org>
2019-11-04runtime: clean up power-of-two rounding code with align functionsMichael Anthony Knyszek
This change renames the "round" function to the more appropriately named "alignUp" which rounds an integer up to the next multiple of a power of two. This change also adds the alignDown function, which is almost like alignUp but rounds down to the previous multiple of a power of two. With these two functions, we also go and replace manual rounding code with it where we can. Change-Id: Ie1487366280484dcb2662972b01b4f7135f72fec Reviewed-on: https://go-review.googlesource.com/c/go/+/190618 Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Keith Randall <khr@golang.org>
2019-07-30runtime: add physHugePageShiftMichael Anthony Knyszek
This change adds physHugePageShift which is defined such that 1 << physHugePageShift == physHugePageSize. The purpose of this variable is to avoid doing expensive divisions in key functions, such as (*mspan).hugePages. This change also does a sweep of any place we might do a division or mod operation with physHugePageSize and turns it into bit shifts and other bitwise operations. Finally, this change adds a check to mallocinit which ensures that physHugePageSize is always a power of two. osinit might choose to ignore non-powers-of-two for the value and replace it with zero, but mallocinit will fail if it's not a power of two (or zero). It also derives physHugePageShift from physHugePageSize. This change helps improve the performance of most applications because of how often (*mspan).hugePages is called. Updates #32828. Change-Id: I1a6db113d52d563f59ae8fd4f0e130858859e68f Reviewed-on: https://go-review.googlesource.com/c/go/+/186598 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-07-29runtime: use uintptr instead of int32 for counting to next heap profile sampleKeith Randall
Overflow of the comparison caused very large (>=1<<32) allocations to sometimes not get sampled at all. Use uintptr so the comparison will never overflow. Fixes #33342 Tested on the example in 33342. I don't want to check a test in that needs that much memory, however. Change-Id: I51fe77a9117affed8094da93c0bc5f445ac2d3d3 Reviewed-on: https://go-review.googlesource.com/c/go/+/188017 Run-TryBot: Keith Randall <khr@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-07-19runtime: align allocations harder in GODEBUG=sbrk=1 modeAustin Clements
Currently, GODEBUG=sbrk=1 mode aligns allocations by their type's alignment. You would think this would be the right thing to do, but because 64-bit fields are only 4-byte aligned right now (see #599), this can cause a 64-bit field of an allocated object to be 4-byte aligned, but not 8-byte aligned. If there is an atomic access to that unaligned 64-bit field, it will crash. This doesn't happen in normal allocation mode because the size-segregated allocation and the current size classes will cause any types larger than 8 bytes to be 8 byte aligned. We fix this by making sbrk=1 mode use alignment based on the type's size rather than its declared alignment. This matches how the tiny allocator aligns allocations. This was tested with GOARCH=386 GODEBUG=sbrk=1 go test sync/atomic This crashes with an unaligned access before this change, and passes with this change. This should be reverted when/if we fix #599. Fixes #33159. Change-Id: Ifc52c72c6b99c5d370476685271baa43ad907565 Reviewed-on: https://go-review.googlesource.com/c/go/+/186919 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Keith Randall <khr@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-05-16runtime: mark newly-mapped memory as scavengedMichael Anthony Knyszek
On most platforms newly-mapped memory is untouched, meaning the pages backing the region haven't been faulted in yet. However, we mark this memory as unscavenged which means the background scavenger aggressively "returns" this memory to the OS if the heap is small. The only platform where newly-mapped memory is actually unscavenged (and counts toward the application's RSS) is on Windows, since (*mheap).sysAlloc commits the reservation. Instead of making a special case for Windows, I change the requirements a bit for a sysReserve'd region. It must now be both sysMap'd and sysUsed'd, with sysMap being a no-op on Windows. Comments about memory allocation have been updated to include a more up-to-date mental model of which states a region of memory may be in (at a very low level) and how to transition between these states. Now this means we can correctly mark newly-mapped heap memory as scavenged on every platform, reducing the load on the background scavenger early on in the application for small heaps. As a result, heap-growth scavenging is no longer necessary, since any actual RSS growth will be accounted for on the allocation codepath. Finally, this change also cleans up grow a little bit to avoid pretending that it's freeing an in-use span and just does the necessary operations directly. Fixes #32012. Fixes #31966. Updates #26473. Change-Id: Ie06061eb638162e0560cdeb0b8993d94cfb4d290 Reviewed-on: https://go-review.googlesource.com/c/go/+/177097 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-05-03runtime: add physHugePageSizeMichael Anthony Knyszek
This change adds the global physHugePageSize which is initialized in osinit(). physHugePageSize contains the system's transparent huge page (or superpage) size in bytes. For #30333. Change-Id: I2f0198c40729dbbe6e6f2676cef1d57dd107562c Reviewed-on: https://go-review.googlesource.com/c/go/+/170858 Run-TryBot: Michael Knyszek <mknyszek@google.com> Reviewed-by: Austin Clements <austin@google.com>
2019-04-16runtime, cmd/link: optimize memory allocation on wasmRichard Musiol
WebAssembly's memory is contiguous. Allocating memory at a high address also allocates all memory up to that address. This change reduces the initial memory allocated on wasm from 1GB to 16MB by using multiple heap arenas and reducing the size of a heap arena. Fixes #27462. Change-Id: Ic941e6edcadd411e65a14cb2f9fd6c8eae02fc7a Reviewed-on: https://go-review.googlesource.com/c/go/+/170950 Run-TryBot: Richard Musiol <neelance@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-25runtime: remove kindNoPointersKeith Randall
We already have the ptrdata field in a type, which encodes exactly the same information that kindNoPointers does. My problem with kindNoPointers is that it often leads to double-negative code like: t.kind & kindNoPointers != 0 Much clearer is: t.ptrdata == 0 Update #27167 Change-Id: I92307d7f018a6bbe3daca4a4abb4225e359349b1 Reviewed-on: https://go-review.googlesource.com/c/go/+/169157 Run-TryBot: Keith Randall <khr@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2019-03-06runtime: align first persistentalloc chunk as requestedIan Lance Taylor
Change-Id: Ib391e019b1a7513d234fb1c8ff802efe8fa7c950 Reviewed-on: https://go-review.googlesource.com/c/go/+/163859 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-02-27internal/reflectlite: lite version of reflect packageMarcel van Lohuizen
to be used by errors package for checking assignability and setting error values in As. Updates #29934. Change-Id: I8c1d02a2c6efa0919d54b286cfe8b4edc26da059 Reviewed-on: https://go-review.googlesource.com/c/161759 Run-TryBot: Marcel van Lohuizen <mpvl@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Russ Cox <rsc@golang.org>
2019-01-18runtime: sample large heap allocations correctlyRaul Silvera
Remove an unnecessary check on the heap sampling code that forced sampling of all heap allocations larger than the sampling rate. This need to follow a poisson process so that they can be correctly unsampled. Maintain a check for MemProfileRate==1 to provide a mechanism for full sampling, as documented in https://golang.org/pkg/runtime/#pkg-variables. Additional testing for this change is on cl/129117. Fixes #26618 Change-Id: I7802bde2afc655cf42cffac34af9bafeb3361957 GitHub-Last-Rev: 471f747af845395d458096bea26daa93b91120be GitHub-Pull-Request: golang/go#29791 Reviewed-on: https://go-review.googlesource.com/c/158337 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
2019-01-04runtime: skip writes to persistent memory in cgo checkerIan Lance Taylor
Fixes #23899 Fixes #28458 Change-Id: Ie177f2d4c399445d8d5e1a327f2419c7866cb45e Reviewed-on: https://go-review.googlesource.com/c/155697 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2018-11-26runtime: handle 64bits addresses for AIXClément Chigot
This commit allows the runtime to handle 64bits addresses returned by mmap syscall on AIX. Mmap syscall returns addresses on 59bits on AIX. But the Arena implementation only allows addresses with less than 48 bits. This commit increases the arena size up to 1<<60 for aix/ppc64. Update: #25893 Change-Id: Iea72e8a944d10d4f00be915785e33ae82dd6329e Reviewed-on: https://go-review.googlesource.com/c/138736 Reviewed-by: Austin Clements <austin@google.com>
2018-11-15runtime: track all heap arenas in a sliceAustin Clements
Currently, there's no efficient way to iterate over the Go heap. We're going to need this for fast free page sweeping, so this CL adds a slice of all allocated heap arenas. This will also be useful for generational GC. For #18155. Change-Id: I58d126cfb9c3f61b3125d80b74ccb1b2169efbcc Reviewed-on: https://go-review.googlesource.com/c/138076 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-10-23runtime: use multiplication with overflow check for newarrayMartin Möhrmann
This improves performance for e.g. maps with a bucket size (key+value*8 bytes) larger than 32 bytes and removes loading a value from the maxElems array for smaller bucket sizes. name old time/op new time/op delta MakeMap/[Byte]Byte 95.5ns ± 1% 94.7ns ± 1% -0.78% (p=0.013 n=9+9) MakeMap/[Int]Int 128ns ± 0% 121ns ± 2% -5.63% (p=0.000 n=6+10) Updates #21588 Change-Id: I7d9eb7d49150c399c15dcab675e24bc97ff97852 Reviewed-on: https://go-review.googlesource.com/c/143997 Reviewed-by: Keith Randall <khr@golang.org>
2018-10-17runtime: use only treaps for tracking spansMichael Anthony Knyszek
Currently, mheap tracks spans in both mSpanLists and mTreaps, but mSpanLists, while they tend to be smaller, complicate the implementation. Here we simplify the implementation by removing free and busy from mheap and renaming freelarge -> free and busylarge -> busy. This change also slightly changes the reclamation policy. Previously, for allocations under 1MB we would attempt to find a small span of the right size. Now, we just try to find any number of spans totaling the right size. This may increase heap fragmentation, but that will be dealt with using virtual memory tricks in follow-up CLs. For #14045. Garbage-heavy benchmarks show very little change, except what appears to be a decrease in STW times and peak RSS. name old STW-ns/GC new STW-ns/GC delta Garbage/benchmem-MB=64-8 263k ±64% 217k ±24% -17.66% (p=0.028 n=25+23) name old STW-ns/op new STW-ns/op delta Garbage/benchmem-MB=64-8 9.39k ±65% 7.80k ±24% -16.88% (p=0.037 n=25+23) name old peak-RSS-bytes new peak-RSS-bytes delta Garbage/benchmem-MB=64-8 281M ± 0% 249M ± 4% -11.40% (p=0.000 n=19+18) https://perf.golang.org/search?q=upload:20181005.1 Go1 benchmarks perform roughly the same, the most notable regression being the JSON encode/decode benchmark with worsens by ~2%. name old time/op new time/op delta BinaryTree17-8 3.02s ± 2% 2.99s ± 2% -1.18% (p=0.000 n=25+24) Fannkuch11-8 3.05s ± 1% 3.02s ± 2% -1.20% (p=0.000 n=25+25) FmtFprintfEmpty-8 43.6ns ± 5% 43.4ns ± 3% ~ (p=0.528 n=25+25) FmtFprintfString-8 74.9ns ± 3% 73.4ns ± 1% -2.03% (p=0.001 n=25+24) FmtFprintfInt-8 79.3ns ± 3% 77.9ns ± 1% -1.73% (p=0.003 n=25+25) FmtFprintfIntInt-8 119ns ± 6% 116ns ± 0% -2.68% (p=0.000 n=25+18) FmtFprintfPrefixedInt-8 134ns ± 4% 132ns ± 1% -1.52% (p=0.004 n=25+25) FmtFprintfFloat-8 240ns ± 1% 241ns ± 1% ~ (p=0.403 n=24+23) FmtManyArgs-8 543ns ± 1% 537ns ± 1% -1.00% (p=0.000 n=25+25) GobDecode-8 6.88ms ± 1% 6.92ms ± 4% ~ (p=0.088 n=24+22) GobEncode-8 5.92ms ± 1% 5.93ms ± 1% ~ (p=0.898 n=25+24) Gzip-8 267ms ± 2% 266ms ± 2% ~ (p=0.213 n=25+24) Gunzip-8 35.4ms ± 1% 35.6ms ± 1% +0.70% (p=0.000 n=25+25) HTTPClientServer-8 104µs ± 2% 104µs ± 2% ~ (p=0.686 n=25+25) JSONEncode-8 9.67ms ± 1% 9.80ms ± 4% +1.32% (p=0.000 n=25+25) JSONDecode-8 47.7ms ± 1% 48.8ms ± 5% +2.33% (p=0.000 n=25+25) Mandelbrot200-8 4.87ms ± 1% 4.91ms ± 1% +0.79% (p=0.000 n=25+25) GoParse-8 3.59ms ± 4% 3.55ms ± 1% ~ (p=0.199 n=25+24) RegexpMatchEasy0_32-8 90.3ns ± 1% 89.9ns ± 1% -0.47% (p=0.000 n=25+21) RegexpMatchEasy0_1K-8 204ns ± 1% 204ns ± 1% ~ (p=0.914 n=25+24) RegexpMatchEasy1_32-8 84.9ns ± 0% 84.6ns ± 1% -0.36% (p=0.000 n=24+25) RegexpMatchEasy1_1K-8 350ns ± 1% 348ns ± 3% -0.59% (p=0.007 n=25+25) RegexpMatchMedium_32-8 122ns ± 1% 121ns ± 0% -1.08% (p=0.000 n=25+18) RegexpMatchMedium_1K-8 36.1µs ± 1% 34.6µs ± 1% -4.02% (p=0.000 n=25+25) RegexpMatchHard_32-8 1.69µs ± 2% 1.65µs ± 1% -2.38% (p=0.000 n=25+25) RegexpMatchHard_1K-8 50.8µs ± 1% 49.4µs ± 1% -2.69% (p=0.000 n=25+24) Revcomp-8 453ms ± 2% 449ms ± 3% -0.74% (p=0.022 n=25+24) Template-8 63.2ms ± 2% 63.4ms ± 1% ~ (p=0.127 n=25+24) TimeParse-8 313ns ± 1% 315ns ± 3% ~ (p=0.924 n=24+25) TimeFormat-8 294ns ± 1% 292ns ± 2% -0.65% (p=0.004 n=23+24) [Geo mean] 49.9µs 49.6µs -0.65% name old speed new speed delta GobDecode-8 112MB/s ± 1% 110MB/s ± 4% -1.00% (p=0.036 n=24+24) GobEncode-8 130MB/s ± 1% 129MB/s ± 1% ~ (p=0.894 n=25+24) Gzip-8 72.7MB/s ± 2% 73.0MB/s ± 2% ~ (p=0.208 n=25+24) Gunzip-8 549MB/s ± 1% 545MB/s ± 1% -0.70% (p=0.000 n=25+25) JSONEncode-8 201MB/s ± 1% 198MB/s ± 3% -1.29% (p=0.000 n=25+25) JSONDecode-8 40.7MB/s ± 1% 39.8MB/s ± 5% -2.23% (p=0.000 n=25+25) GoParse-8 16.2MB/s ± 4% 16.3MB/s ± 1% ~ (p=0.211 n=25+24) RegexpMatchEasy0_32-8 354MB/s ± 1% 356MB/s ± 1% +0.47% (p=0.000 n=25+21) RegexpMatchEasy0_1K-8 5.00GB/s ± 0% 4.99GB/s ± 1% ~ (p=0.588 n=24+24) RegexpMatchEasy1_32-8 377MB/s ± 1% 378MB/s ± 1% +0.39% (p=0.000 n=25+25) RegexpMatchEasy1_1K-8 2.92GB/s ± 1% 2.94GB/s ± 3% +0.65% (p=0.008 n=25+25) RegexpMatchMedium_32-8 8.14MB/s ± 1% 8.22MB/s ± 1% +0.98% (p=0.000 n=25+24) RegexpMatchMedium_1K-8 28.4MB/s ± 1% 29.6MB/s ± 1% +4.19% (p=0.000 n=25+25) RegexpMatchHard_32-8 18.9MB/s ± 2% 19.4MB/s ± 1% +2.43% (p=0.000 n=25+25) RegexpMatchHard_1K-8 20.2MB/s ± 1% 20.7MB/s ± 1% +2.76% (p=0.000 n=25+24) Revcomp-8 561MB/s ± 2% 566MB/s ± 3% +0.75% (p=0.021 n=25+24) Template-8 30.7MB/s ± 2% 30.6MB/s ± 1% ~ (p=0.131 n=25+24) [Geo mean] 120MB/s 121MB/s +0.48% https://perf.golang.org/search?q=upload:20181004.6 Change-Id: I97f9fee34577961a116a8ddd445c6272253f0f95 Reviewed-on: https://go-review.googlesource.com/c/139837 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2018-10-04runtime: clarify table of arena sizesAustin Clements
Currently the table of arena sizes mixes the number of entries in the L1 with the size of the L2. While the size of the L2 is important, this makes it hard to see what's actually going on because there's an implicit factor of sys.PtrSize. This changes the L2 column to say both the number of entries and the size that results in. This should hopefully make the relations between the columns of the table clearer, since they can now be plugged directly into the given formula. Change-Id: Ie677adaef763b893a2f620bd4fc3b8db314b3a1e Reviewed-on: https://go-review.googlesource.com/c/139697 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-02runtime: remove gcStart's mode argumentAustin Clements
This argument is always gcBackgroundMode since only debug.gcstoptheworld can trigger a STW GC at this point. Remove the unnecessary argument. Updates #26903. This is preparation for unifying STW GC and concurrent GC. Change-Id: Icb4ba8f10f80c2b69cf51a21e04fa2c761b71c94 Reviewed-on: https://go-review.googlesource.com/c/134775 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-09-28runtime: don't call mcache.refill on systemstackAustin Clements
mcache.refill doesn't need to run on the system stack; it just needs to be non-preemptible. Its only caller, mcache.nextFree, also needs to be non-preemptible, so we can remove the unnecessary systemstack switch. Change-Id: Iba5b3f4444855f1dc134485ba588efff3b54c426 Reviewed-on: https://go-review.googlesource.com/138196 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Rick Hudson <rlh@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-09-26runtime: rename _MSpan* constants to mSpan*Austin Clements
We already aliased mSpanInUse to _MSpanInUse. The dual constants are getting annoying, so fix all of these to use the mSpan* naming convention. This was done automatically with: sed -i -re 's/_?MSpan(Dead|InUse|Manual|Free)/mSpan\1/g' *.go plus deleting the existing definition of mSpanInUse. Change-Id: I09979d9d491d06c10689cea625dc57faa9cc6767 Reviewed-on: https://go-review.googlesource.com/137875 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-09-25runtime: fix reference to sys{Fault,Free,Reserve,Unused,Used} in commentsTobias Klauser
Change-Id: Icbaedc49c810c63c51d56ae394d2f70e4d64b3e0 Reviewed-on: https://go-review.googlesource.com/136495 Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
2018-07-19runtime: don't create heap hints outside TSAN's supported heapAustin Clements
TSAN for Go only supports heap address in the range [0x00c000000000, 0x00e000000000). However, we currently create heap hints of the form 0xXXc000000000 for XX between 0x00 and 0x7f. Even for XX=0x01, this hint is outside TSAN's supported heap address range. Fix this by creating a slightly different set of hints in race mode, all of which fall inside TSAN's heap address range. This should fix TestArenaCollision flakes. That test forces the runtime to use later heap hints. Currently, this always results in TSAN "failed to allocate" failures on Windows (which happens to have a slightly more constrained TSAN layout than non-Windows). Most of the time we don't notice these failures, but sometimes it crashes TSAN, leading to a test failure. Fixes #25698. Change-Id: I8926cd61f0ee5ee00efa77b283f7b809c555be46 Reviewed-on: https://go-review.googlesource.com/123780 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-05-08runtime: add js/wasm architectureRichard Musiol
This commit adds the js/wasm architecture to the runtime package. Currently WebAssembly has no support for threads yet, see https://github.com/WebAssembly/design/issues/1073. Because of that, there is no preemption of goroutines and no sysmon goroutine. Design doc: https://docs.google.com/document/d/131vjr4DH6JFnb-blm_uRdaC0_Nv3OUwjEY5qVCxCup4 About WebAssembly assembly files: https://docs.google.com/document/d/1GRmy3rA4DiYtBlX-I1Jr_iHykbX8EixC3Mq0TCYqbKc Updates #18892 Change-Id: I7f12d21b5180500d55ae9fd2f7e926a1731db391 Reviewed-on: https://go-review.googlesource.com/103877 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2018-04-04runtime: stop when we run out of hints in race modeAustin Clements
Currently, the runtime falls back to asking for any address the OS can offer for the heap when it runs out of hint addresses. However, the race detector assumes the heap lives in [0x00c000000000, 0x00e000000000), and will fail in a non-obvious way if we go outside this region. Fix this by actively throwing a useful error if we run out of heap hints in race mode. This problem is currently being triggered by TestArenaCollision, which intentionally triggers this fallback behavior. Fix the test to look for the new panic message in race mode. Fixes #24670. Updates #24133. Change-Id: I57de6d17a3495dc1f1f84afc382cd18a6efc2bf7 Reviewed-on: https://go-review.googlesource.com/104717 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
2018-02-23runtime: reduce arena size to 4MB on 64-bit WindowsAustin Clements
Currently, we use 64MB heap arenas on 64-bit platforms. This works well on UNIX-like OSes because they treat untouched pages as essentially free. However, on Windows, committed memory is charged against a process whether or not it has demand-faulted physical pages in. Hence, on Windows, even a process with a tiny heap will commit 64MB for one heap arena, plus another 32MB for the arena map. Things are much worse under the race detector, which increases the heap commitment by a factor of 5.5X, leading to 384MB of committed memory at runtime init. Fix this by reducing the heap arena size to 4MB on Windows. To counterbalance the effect of increasing the arena map size by a factor of 16, and to further reduce the impact of the commitment for the arena map, we switch from a single entry L1 arena map to a 64 entry L1 arena map. Compared to the original arena design, this slows down the x/benchmarks garbage benchmark by 0.49% (the slow down of this commit alone is 1.59%, but the previous commit bought us a 1% speed-up): name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.28ms ± 1% 2.29ms ± 1% +0.49% (p=0.000 n=17+18) (https://perf.golang.org/search?q=upload:20180223.1) (This was measured on linux/amd64 by modifying its arena configuration as above.) Fixes #23900. Change-Id: I6b7fa5ecebee2947bf20cfeb78c248809469c6b1 Reviewed-on: https://go-review.googlesource.com/96780 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-23runtime: support a two-level arena mapAustin Clements
Currently, the heap arena map is a single, large array that covers every possible arena frame in the entire address space. This is practical up to about 48 bits of address space with 64 MB arenas. However, there are two problems with this: 1. mips64, ppc64, and s390x support full 64-bit address spaces (though on Linux only s390x has kernel support for 64-bit address spaces). On these platforms, it would be good to support these larger address spaces. 2. On Windows, processes are charged for untouched memory, so for processes with small heaps, the mostly-untouched 32 MB arena map plus a 64 MB arena are significant overhead. Hence, it would be good to reduce both the arena map size and the arena size, but with a single-level arena, these are inversely proportional. This CL adds support for a two-level arena map. Arena frame numbers are now divided into arenaL1Bits of L1 index and arenaL2Bits of L2 index. At the moment, arenaL1Bits is always 0, so we effectively have a single level map. We do a few things so that this has no cost beyond the current single-level map: 1. We embed the L2 array directly in mheap, so if there's a single entry in the L2 array, the representation is identical to the current representation and there's no extra level of indirection. 2. Hot code that accesses the arena map is structured so that it optimizes to nearly the same machine code as it does currently. 3. We make some small tweaks to hot code paths and to the inliner itself to keep some important functions inlined despite their now-larger ASTs. In particular, this is necessary for heapBitsForAddr and heapBits.next. Possibly as a result of some of the tweaks, this actually slightly improves the performance of the x/benchmarks garbage benchmark: name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.28ms ± 1% 2.26ms ± 1% -1.07% (p=0.000 n=17+19) (https://perf.golang.org/search?q=upload:20180223.2) For #23900. Change-Id: If5164e0961754f97eb9eca58f837f36d759505ff Reviewed-on: https://go-review.googlesource.com/96779 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-23runtime: rename "arena index" to "arena map"Austin Clements
There are too many places where I want to talk about "indexing into the arena index". Make this less awkward and ambiguous by calling it the "arena map" instead. Change-Id: I726b0667bb2139dbc006175a0ec09a871cdf73f9 Reviewed-on: https://go-review.googlesource.com/96777 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-21runtime: clarify address space limit constants and commentsAustin Clements
Now that we support the full non-contiguous virtual address space of amd64 hardware, some of the comments and constants related to this are out of date. This renames memLimitBits to heapAddrBits because 1<<memLimitBits is no longer the limit of the address space and rewrites the comment to focus first on hardware limits (which span OSes) and then discuss kernel limits. Second, this eliminates the memLimit constant because there's no longer a meaningful "highest possible heap pointer value" on amd64. Updates #23862. Change-Id: I44b32033d2deb6b69248fb8dda14fc0e65c47f11 Reviewed-on: https://go-review.googlesource.com/95498 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-21runtime: offset the heap arena index by 2^47 on amd64Austin Clements
On amd64, the virtual address space, when interpreted as signed values, is [-2^47, 2^47). Currently, we only support heap addresses in the "positive" half of this, [0, 2^47). This suffices for linux/amd64 and windows/amd64, but solaris/amd64 can map user addresses in the negative part of this range. Specifically, addresses 0xFFFF8000'00000000 to 0xFFFFFD80'00000000 are part of user space. This leads to "memory allocated by OS not in usable address space" panic, since we don't map heap arena index space for these addresses. Fix this by offsetting addresses when computing arena indexes so that arena entry 0 corresponds to address -2^47 on amd64. We already map enough arena space for 2^48 heap addresses on 64-bit (because arm64's virtual address space is [0, 2^48)), so we don't need to grow any structures to support this. A different approach would be to simply mask out the top 16 bits. However, there are two advantages to the offset approach: 1) invalid heap addresses continue to naturally map to invalid arena indexes so we don't need extra checks and 2) it perturbs the mapping of addresses to arena indexes more, which helps check that we don't accidentally compute incorrect arena indexes somewhere that happen to be right most of the time. Several comments and constant names are now somewhat misleading. We'll fix that in the next CL. This CL is the core change the arena indexing. Fixes #23862. Change-Id: Idb8e299fded04593a286b01a9582da6ddbac2f9a Reviewed-on: https://go-review.googlesource.com/95497 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-21runtime: abstract indexing of arena indexAustin Clements
Accessing the arena index is about to get slightly more complicated. Abstract this away into a set of functions for going back and forth between addresses and arena slice indexes. For #23862. Change-Id: I0b20e74ef47a07b78ed0cf0a6128afe6f6e40f4b Reviewed-on: https://go-review.googlesource.com/95496 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-21runtime: ensure sysStat for mheap_.arenas is alignedAustin Clements
We don't want to account the memory for mheap_.arenas because most of it is never touched, so currently we pass the address of a uint64 on the heap. However, at least on mips, it's possible for this uint64 to be unaligned, which causes the atomic add in mSysStatInc to crash. Fix this by instead passing a nil stat pointer. Fixes #23946. Change-Id: I091587df1b3066c330b6bb4d834e4596c407910f Reviewed-on: https://go-review.googlesource.com/95695 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2018-02-21runtime: shorten reflect.unsafe_New call chainMartin Möhrmann
reflect.unsafe_New is an often called function according to profiling in a large production environment. Since newobject is not inlined currently there is call overhead that can be avoided by calling mallocgc directly. name old time/op new time/op delta New 32.4ns ± 2% 29.8ns ± 1% -8.03% (p=0.000 n=19+20) Change-Id: I572e4be830ed8e5c0da555dc3a8864c8363112be Reviewed-on: https://go-review.googlesource.com/95015 Reviewed-by: Austin Clements <austin@google.com>
2018-02-15runtime: replace _MaxMem with maxAllocAustin Clements
Now that we have memLimit, also having _MaxMem is a bit confusing. Replace it with maxAlloc, which better conveys what it limits. We also define maxAlloc slightly differently: since it's now clear that it limits allocation size, we can account for a subtle difference between 32-bit and 64-bit. Change-Id: Iac39048018cc0dae7f0919e25185fee4b3eed529 Reviewed-on: https://go-review.googlesource.com/85890 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-15runtime: move comment about address space sizes to malloc.goAustin Clements
Currently there's a detailed comment in lfstack_64bit.go about address space limitations on various architectures. Since that's now relevant to malloc, move it to a more prominent place in the documentation for memLimitBits. Updates #10460. Change-Id: If9708291cf3a288057b8b3ba0ba6a59e3602bbd6 Reviewed-on: https://go-review.googlesource.com/85889 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-15runtime: remove non-reserved heap logicAustin Clements
Currently large sysReserve calls on some OSes don't actually reserve the memory, but just check that it can be reserved. This was important when we called sysReserve to "reserve" many gigabytes for the heap up front, but now that we map memory in small increments as we need it, this complication is no longer necessary. This has one curious side benefit: currently, on Linux, allocations that are large enough to be rejected by mmap wind up freezing the application for a long time before it panics. This happens because sysReserve doesn't reserve the memory, so sysMap calls mmap_fixed, which calls mmap, which fails because the mapping is too large. However, mmap_fixed doesn't inspect *why* mmap fails, so it falls back to probing every page in the desired region individually with mincore before performing an (otherwise dangerous) MAP_FIXED mapping, which will also fail. This takes a long time for a large region. Now this logic is gone, so the mmap failure leads to an immediate panic. Updates #10460. Change-Id: I8efe88c611871cdb14f99fadd09db83e0161ca2e Reviewed-on: https://go-review.googlesource.com/85888 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-15runtime: use sparse mappings for the heapAustin Clements
This replaces the contiguous heap arena mapping with a potentially sparse mapping that can support heap mappings anywhere in the address space. This has several advantages over the current approach: * There is no longer any limit on the size of the Go heap. (Currently it's limited to 512GB.) Hence, this fixes #10460. * It eliminates many failures modes of heap initialization and growing. In particular it eliminates any possibility of panicking with an address space conflict. This can happen for many reasons and even causes a low but steady rate of TSAN test failures because of conflicts with the TSAN runtime. See #16936 and #11993. * It eliminates the notion of "non-reserved" heap, which was added because creating huge address space reservations (particularly on 64-bit) led to huge process VSIZE. This was at best confusing and at worst conflicted badly with ulimit -v. However, the non-reserved heap logic is complicated, can race with other mappings in non-pure Go binaries (e.g., #18976), and requires that the entire heap be either reserved or non-reserved. We currently maintain the latter property, but it's quite difficult to convince yourself of that, and hence difficult to keep correct. This logic is still present, but will be removed in the next CL. * It fixes problems on 32-bit where skipping over parts of the address space leads to mapping huge (and never-to-be-used) metadata structures. See #19831. This also completely rewrites and significantly simplifies mheap.sysAlloc, which has been a source of many bugs. E.g., #21044, #20259, #18651, and #13143 (and maybe #23222). This change also makes it possible to allocate individual objects larger than 512GB. As a result, a few tests that expected huge allocations to fail needed to be changed to make even larger allocations. However, at the moment attempting to allocate a humongous object may cause the program to freeze for several minutes on Linux as we fall back to probing every page with addrspace_free. That logic (and this failure mode) will be removed in the next CL. Fixes #10460. Fixes #22204 (since it rewrites the code involved). This slightly slows down compilebench and the x/benchmarks garbage benchmark. name old time/op new time/op delta Template 184ms ± 1% 185ms ± 1% ~ (p=0.065 n=10+9) Unicode 86.9ms ± 3% 86.3ms ± 1% ~ (p=0.631 n=10+10) GoTypes 599ms ± 0% 602ms ± 0% +0.56% (p=0.000 n=10+9) Compiler 2.87s ± 1% 2.89s ± 1% +0.51% (p=0.002 n=9+10) SSA 7.29s ± 1% 7.25s ± 1% ~ (p=0.182 n=10+9) Flate 118ms ± 2% 118ms ± 1% ~ (p=0.113 n=9+9) GoParser 147ms ± 1% 148ms ± 1% +1.07% (p=0.003 n=9+10) Reflect 401ms ± 1% 404ms ± 1% +0.71% (p=0.003 n=10+9) Tar 175ms ± 1% 175ms ± 1% ~ (p=0.604 n=9+10) XML 209ms ± 1% 210ms ± 1% ~ (p=0.052 n=10+10) (https://perf.golang.org/search?q=upload:20171231.4) name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.23ms ± 1% 2.25ms ± 1% +0.84% (p=0.000 n=19+19) (https://perf.golang.org/search?q=upload:20171231.3) Relative to the start of the sparse heap changes (starting at and including "runtime: fix various contiguous bitmap assumptions"), overall slowdown is roughly 1% on GC-intensive benchmarks: name old time/op new time/op delta Template 183ms ± 1% 185ms ± 1% +1.32% (p=0.000 n=9+9) Unicode 84.9ms ± 2% 86.3ms ± 1% +1.65% (p=0.000 n=9+10) GoTypes 595ms ± 1% 602ms ± 0% +1.19% (p=0.000 n=9+9) Compiler 2.86s ± 0% 2.89s ± 1% +0.91% (p=0.000 n=9+10) SSA 7.19s ± 0% 7.25s ± 1% +0.75% (p=0.000 n=8+9) Flate 117ms ± 1% 118ms ± 1% +1.10% (p=0.000 n=10+9) GoParser 146ms ± 2% 148ms ± 1% +1.48% (p=0.002 n=10+10) Reflect 398ms ± 1% 404ms ± 1% +1.51% (p=0.000 n=10+9) Tar 173ms ± 1% 175ms ± 1% +1.17% (p=0.000 n=10+10) XML 208ms ± 1% 210ms ± 1% +0.62% (p=0.011 n=10+10) [Geo mean] 369ms 373ms +1.17% (https://perf.golang.org/search?q=upload:20180101.2) name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.22ms ± 1% 2.25ms ± 1% +1.51% (p=0.000 n=20+19) (https://perf.golang.org/search?q=upload:20180101.3) Change-Id: I5daf4cfec24b252e5a57001f0a6c03f22479d0f0 Reviewed-on: https://go-review.googlesource.com/85887 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-15runtime: eliminate most uses of mheap_.arena_*Austin Clements
This replaces all uses of the mheap_.arena_* fields outside of mallocinit and sysAlloc. These fields fundamentally assume a contiguous heap between two bounds, so eliminating these is necessary for a sparse heap. Many of these are replaced with checks for non-nil spans at the test address (which in turn checks for a non-nil entry in the heap arena array). Some of them are just for debugging and somewhat meaningless with a sparse heap, so those we just delete. Updates #10460. Change-Id: I8345b95ffc610aed694f08f74633b3c63506a41f Reviewed-on: https://go-review.googlesource.com/85886 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-15runtime: make span map sparseAustin Clements
This splits the span map into separate chunks for every 64MB of the heap. The span map chunks now live in the same indirect structure as the bitmap. Updates #10460. This causes a slight improvement in compilebench and the x/benchmarks garbage benchmark. I'm not sure why it improves performance. name old time/op new time/op delta Template 185ms ± 1% 184ms ± 1% ~ (p=0.315 n=9+10) Unicode 86.9ms ± 1% 86.9ms ± 3% ~ (p=0.356 n=9+10) GoTypes 602ms ± 1% 599ms ± 0% -0.59% (p=0.002 n=9+10) Compiler 2.89s ± 0% 2.87s ± 1% -0.50% (p=0.003 n=9+9) SSA 7.25s ± 0% 7.29s ± 1% ~ (p=0.400 n=9+10) Flate 118ms ± 1% 118ms ± 2% ~ (p=0.065 n=10+9) GoParser 147ms ± 2% 147ms ± 1% ~ (p=0.549 n=10+9) Reflect 403ms ± 1% 401ms ± 1% -0.47% (p=0.035 n=9+10) Tar 176ms ± 1% 175ms ± 1% -0.59% (p=0.013 n=10+9) XML 211ms ± 1% 209ms ± 1% -0.83% (p=0.011 n=10+10) (https://perf.golang.org/search?q=upload:20171231.1) name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.24ms ± 1% 2.23ms ± 1% -0.36% (p=0.001 n=20+19) (https://perf.golang.org/search?q=upload:20171231.2) Change-Id: I2563f8704ab9812434947faf293c5327f9b0d07a Reviewed-on: https://go-review.googlesource.com/85885 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-15runtime: make the heap bitmap sparseAustin Clements
This splits the heap bitmap into separate chunks for every 64MB of the heap and introduces an index mapping from virtual address to metadata. It modifies the heapBits abstraction to use this two-level structure. Finally, it modifies heapBitsSetType to unroll the bitmap into the object itself and then copy it out if the bitmap would span discontiguous bitmap chunks. This is a step toward supporting general sparse heaps, which will eliminate address space conflict failures as well as the limit on the heap size. It's also advantageous for 32-bit. 32-bit already supports discontiguous heaps by always starting the arena at address 0. However, as a result, with a contiguous bitmap, if the kernel chooses a high address (near 2GB) for a heap mapping, the runtime is forced to map up to 128MB of heap bitmap. Now the runtime can map sections of the bitmap for just the parts of the address space used by the heap. Updates #10460. This slightly slows down the x/garbage and compilebench benchmarks. However, I think the slowdown is acceptably small. name old time/op new time/op delta Template 178ms ± 1% 180ms ± 1% +0.78% (p=0.029 n=10+10) Unicode 85.7ms ± 2% 86.5ms ± 2% ~ (p=0.089 n=10+10) GoTypes 594ms ± 0% 599ms ± 1% +0.70% (p=0.000 n=9+9) Compiler 2.86s ± 0% 2.87s ± 0% +0.40% (p=0.001 n=9+9) SSA 7.23s ± 2% 7.29s ± 2% +0.94% (p=0.029 n=10+10) Flate 116ms ± 1% 117ms ± 1% +0.99% (p=0.000 n=9+9) GoParser 146ms ± 1% 146ms ± 0% ~ (p=0.193 n=10+7) Reflect 399ms ± 0% 403ms ± 1% +0.89% (p=0.001 n=10+10) Tar 173ms ± 1% 174ms ± 1% +0.91% (p=0.013 n=10+9) XML 208ms ± 1% 210ms ± 1% +0.93% (p=0.000 n=10+10) [Geo mean] 368ms 371ms +0.79% name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.17ms ± 1% 2.21ms ± 1% +2.15% (p=0.000 n=20+20) Change-Id: I037fd283221976f4f61249119d6b97b100bcbc66 Reviewed-on: https://go-review.googlesource.com/85883 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2018-02-15runtime: lay out heap bitmap forward in memoryAustin Clements
Currently the heap bitamp is laid in reverse order in memory relative to the heap itself. This was originally done out of "excessive cleverness" so that computing a bitmap pointer could load only the arena_start field and so that heaps could be more contiguous by growing the arena and the bitmap out from a common center point. However, this appears to have no actual performance benefit, it complicates nearly every use of the bitmap, and it makes already confusing code more confusing. Furthermore, it's still possible to use a single field (the new bitmap_delta) for the bitmap pointer computation by employing slightly different excessive cleverness. Hence, this CL puts the bitmap into forward order. This is a (very) updated version of CL 9404. Change-Id: I743587cc626c4ecd81e660658bad85b54584108c Reviewed-on: https://go-review.googlesource.com/85881 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2017-10-29runtime: eliminate write barriers from persistentallocAustin Clements
We're about to start tracking nowritebarrierrec through systemstack calls, which will reveal write barriers in persistentalloc prohibited by various callers. The pointers manipulated by persistentalloc are always to off-heap memory, so this removes these write barriers statically by introducing a new go:notinheap type to represent generic off-heap memory. Updates #22384. For #22460. Change-Id: Id449d9ebf145b14d55476a833e7f076b0d261d57 Reviewed-on: https://go-review.googlesource.com/72771 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
2017-09-26runtime: improve comments for nextSampleGiovanni Bajo
The previous comment of nextSample didn't mention Poisson processes, which is the reason why it needed to create an exponential distribution, so it was hard to follow the reasoning for people not highly familiar with statistics. Since we're at it, we also make it clear that we are just creating a random number with exponential distribution by moving the bulk of the function into a new fastexprand(). No functional changes. Change-Id: I9c275e87edb3418ee0974257af64c73465028ad7 Reviewed-on: https://go-review.googlesource.com/65657 Reviewed-by: Austin Clements <austin@google.com> Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2017-09-20runtime: make nextFreeFast inlinableIlya Tocar
https://golang.org/cl/22598 made nextFreeFast inlinable. But during https://golang.org/cl/63611 it was discovered, that it is no longer inlinable. Reduce number of statements below inlining threshold to make it inlinable again. Also update tests, to prevent regressions. Doesn't reduce readability. Change-Id: Ia672784dd48ed3b1ab46e390132f1094fe453de5 Reviewed-on: https://go-review.googlesource.com/65030 Run-TryBot: Ilya Tocar <ilya.tocar@intel.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
2017-08-14runtime: special case allocation of arrays of size 1Josh Bleecher Snyder
This avoids division and multiplication. Instrumentation suggests that this is a very common case. Change-Id: I2d5d5012d4f4df4c4af1f9f85ca9c323c9889c0e Reviewed-on: https://go-review.googlesource.com/54657 Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2017-07-31runtime: fall back to small mmaps if we fail to grow reservationAustin Clements
Right now, if it's possible to grow the arena reservation but mheap.sysAlloc fails to get 256MB more of memory, it simply fails. However, on 32-bit we have a fallback path that uses much smaller mmaps that could take in this situation, but fail to. This commit fixes mheap.sysAlloc to use a common failure path in case it can't grow the reservation. On 32-bit, this path includes the fallback. Ideally, mheap.sysAlloc would attempt smaller reservation growths first, but taking the fallback path is a simple change for Go 1.9. Updates #21044 (fixes one of two issues). Change-Id: I1e0035ffba986c3551479d5742809e43da5e7c73 Reviewed-on: https://go-review.googlesource.com/51713 Run-TryBot: Austin Clements <austin@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>
generated by cgit v1.3 (git 2.53.0) at 2026-04-15 20:07:37 +0000