runtime: set and clear only the relevant bits in allocToCache

Currently allocToCache ham-handedly calls pageAlloc.allocRange on the
full size of the cache. This is fine as long as scavenged bits are never
set when alloc bits are set. This is true right now, but won't be true
as of the next CL.

This change makes allocToCache more carefully set the bits. Note that in
the allocToCache path, we were also calling update *twice*, erroneously.
The first time, with contig=true! Luckily today there's no correctness
error there because the page cache is small enough that the contig=true
logic doesn't matter, but this should at least improve allocation
performance a little bit.

Change-Id: I3ff9590ac86d251e4c5063cfd633570238b0cdbf
Reviewed-on: https://go-review.googlesource.com/c/go/+/356609
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>

1 files changed, 18 insertions, 0 deletions

diff --git a/src/runtime/mpallocbits.go b/src/runtime/mpallocbits.go
index ff112300c3..f63164becd 100644
--- a/src/runtime/mpallocbits.go
+++ b/src/runtime/mpallocbits.go
@@ -57,6 +57,12 @@ func (b *pageBits) setAll() {
 	}
 }
 
+// setBlock64 sets the 64-bit aligned block of bits containing the i'th bit that
+// are set in v.
+func (b *pageBits) setBlock64(i uint, v uint64) {
+	b[i/64] |= v
+}
+
 // clear clears bit i of pageBits.
 func (b *pageBits) clear(i uint) {
 	b[i/64] &^= 1 << (i % 64)
@@ -93,6 +99,12 @@ func (b *pageBits) clearAll() {
 	}
 }
 
+// clearBlock64 clears the 64-bit aligned block of bits containing the i'th bit that
+// are set in v.
+func (b *pageBits) clearBlock64(i uint, v uint64) {
+	b[i/64] &^= v
+}
+
 // popcntRange counts the number of set bits in the
 // range [i, i+n).
 func (b *pageBits) popcntRange(i, n uint) (s uint) {
@@ -367,6 +379,12 @@ func (b *pallocBits) pages64(i uint) uint64 {
 	return (*pageBits)(b).block64(i)
 }
 
+// allocPages64 allocates a 64-bit block of 64 pages aligned to 64 pages according
+// to the bits set in alloc. The block set is the one containing the i'th page.
+func (b *pallocBits) allocPages64(i uint, alloc uint64) {
+	(*pageBits)(b).setBlock64(i, alloc)
+}
+
 // findBitRange64 returns the bit index of the first set of
 // n consecutive 1 bits. If no consecutive set of 1 bits of
 // size n may be found in c, then it returns an integer >= 64.