cmd/compile: introduce alias analysis and automatically free non-aliased memory after growslice

This CL is part of a set of CLs that attempt to reduce how much work the
GC must do. See the design in https://go.dev/design/74299-runtime-freegc

This CL updates the compiler to examine append calls to prove
whether or not the slice is aliased.

If proven unaliased, the compiler automatically inserts a call to a new
runtime function introduced with this CL, runtime.growsliceNoAlias,
which frees the old backing memory immediately after slice growth is
complete and the old storage is logically dead.

Two append benchmarks below show promising results, executing up to
~2x faster and up to factor of ~3 memory reduction with this CL.

The approach works with multiple append calls for the same slice,
including inside loops, and the final slice memory can be escaping,
such as in a classic pattern of returning a slice from a function
after the slice is built. (The final slice memory is never freed with
this CL, though we have other work that tackles that.)

An example target for this CL is we automatically free the
intermediate memory for the appends in the loop in this function:

    func f1(input []int) []int {
        var s []int
        for _, x := range input {
            s = append(s, g(x))   // s cannot be aliased here
            if h(x) {
                s = append(s, x)  // s cannot be aliased here
            }
        }
        return s                  // slice escapes at end
    }

In this case, the compiler and the runtime collaborate so that
the heap allocated backing memory for s is automatically freed after
a successful grow. (For the first grow, there is nothing to free,
but for the second and subsequent growths, the old heap memory is
freed automatically.)

The new runtime.growsliceNoAlias is primarily implemented
by calling runtime.freegc, which we introduced in CL 673695.

The high-level approach here is we step through the IR starting
from a slice declaration and look for any operations that either
alias the slice or might do so, and treat any IR construct we
don't specifically handle as a potential alias (and therefore
conservatively fall back to treating the slice as aliased when
encountering something not understood).

For loops, some additional care is required. We arrange the analysis
so that an alias in the body of a loop causes all the appends in that
same loop body to be marked aliased, even if the aliasing occurs after
the append in the IR:

    func f2() {
        var s []int
        for i := range 10 {
                s = append(s, i)  // aliased due to next line
                alias = s
        }
    }

For nested loops, we analyse the nesting appropriately so that
for example this append is still proven as non-aliased in the
inner loop even though it aliased for the outer loop:

    func f3() {
        for range 10 {
            var s []int
            for i := range 10 {
                s = append(s, i)  // append using non-aliased slice
            }
            alias = s
        }
    }

A good starting point is the beginning of the test/escape_alias.go file,
which starts with ~10 introductory examples with brief comments that
attempt to illustrate the high-level approach.

For more details, see the new .../internal/escape/alias.go file,
especially the (*aliasAnalysis).analyze method.

In the first benchmark, an append in a loop builds up a slice from
nothing, where the slice elements are each 64 bytes. In the table below,
'count' is the number of appends. With 1 append, there is no opportunity
for this CL to free memory. Once there are 2 appends, the growth from
1 element to 2 elements means the compiler-inserted growsliceNoAlias
frees the 1-element array, and we see a ~33% reduction in memory use
and a small reported speed improvement.

As the number of appends increases for example to 5, we are at
a ~20% speed improvement and ~45% memory reduction, and so on until
we reach ~40% faster and ~50% less memory allocated at the end of
the table.

There can be variation in the reported numbers based on -randlayout, so
this table is for 30 different values of -randlayout with a total
n=150. (Even so, there is still some variation, so we probably should
not read too much into small changes.) This is with GOAMD64=v3 on
a VM that gcc reports is cascadelake.

goos: linux
goarch: amd64
pkg: runtime
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                         │ old-1bb1f2bf0c │        freegc-8ba7421-ps16 │
                         │     sec/op     │   sec/op     vs base       │
Append64Bytes/count=1-4       31.09n ± 2%   31.69n ± 1%   +1.95% (n=150)
Append64Bytes/count=2-4       73.31n ± 1%   70.27n ± 0%   -4.15% (n=150)
Append64Bytes/count=3-4       142.7n ± 1%   124.6n ± 1%  -12.68% (n=150)
Append64Bytes/count=4-4       149.6n ± 1%   127.7n ± 0%  -14.64% (n=150)
Append64Bytes/count=5-4       277.1n ± 1%   213.6n ± 0%  -22.90% (n=150)
Append64Bytes/count=6-4       280.7n ± 1%   216.5n ± 1%  -22.87% (n=150)
Append64Bytes/count=10-4      544.3n ± 1%   386.6n ± 0%  -28.97% (n=150)
Append64Bytes/count=20-4     1058.5n ± 1%   715.6n ± 1%  -32.39% (n=150)
Append64Bytes/count=50-4      2.121µ ± 1%   1.404µ ± 1%  -33.83% (n=150)
Append64Bytes/count=100-4     4.152µ ± 1%   2.736µ ± 1%  -34.11% (n=150)
Append64Bytes/count=200-4     7.753µ ± 1%   4.882µ ± 1%  -37.03% (n=150)
Append64Bytes/count=400-4    15.163µ ± 2%   9.273µ ± 1%  -38.84% (n=150)
geomean                       601.8n        455.0n       -24.39%

                          │ old-1bb1f2bf0c │      freegc-8ba7421-ps16  │
                          │      B/op      │  B/op      vs base        │
Append64Bytes/count=1-4       64.00 ± 0%     64.00 ± 0%        ~ (n=150)
Append64Bytes/count=2-4       192.0 ± 0%     128.0 ± 0%  -33.33% (n=150)
Append64Bytes/count=3-4       448.0 ± 0%     256.0 ± 0%  -42.86% (n=150)
Append64Bytes/count=4-4       448.0 ± 0%     256.0 ± 0%  -42.86% (n=150)
Append64Bytes/count=5-4       960.0 ± 0%     512.0 ± 0%  -46.67% (n=150)
Append64Bytes/count=6-4       960.0 ± 0%     512.0 ± 0%  -46.67% (n=150)
Append64Bytes/count=10-4    1.938Ki ± 0%   1.000Ki ± 0%  -48.39% (n=150)
Append64Bytes/count=20-4    3.938Ki ± 0%   2.001Ki ± 0%  -49.18% (n=150)
Append64Bytes/count=50-4    7.938Ki ± 0%   4.005Ki ± 0%  -49.54% (n=150)
Append64Bytes/count=100-4  15.938Ki ± 0%   8.021Ki ± 0%  -49.67% (n=150)
Append64Bytes/count=200-4   31.94Ki ± 0%   16.08Ki ± 0%  -49.64% (n=150)
Append64Bytes/count=400-4   63.94Ki ± 0%   32.33Ki ± 0%  -49.44% (n=150)
geomean                     1.991Ki        1.124Ki       -43.54%

                          │ old-1bb1f2bf0c │     freegc-8ba7421-ps16   │
                          │   allocs/op    │ allocs/op   vs base       │
Append64Bytes/count=1-4         1.000 ± 0%   1.000 ± 0%        ~ (n=150)
Append64Bytes/count=2-4         2.000 ± 0%   1.000 ± 0%  -50.00% (n=150)
Append64Bytes/count=3-4         3.000 ± 0%   1.000 ± 0%  -66.67% (n=150)
Append64Bytes/count=4-4         3.000 ± 0%   1.000 ± 0%  -66.67% (n=150)
Append64Bytes/count=5-4         4.000 ± 0%   1.000 ± 0%  -75.00% (n=150)
Append64Bytes/count=6-4         4.000 ± 0%   1.000 ± 0%  -75.00% (n=150)
Append64Bytes/count=10-4        5.000 ± 0%   1.000 ± 0%  -80.00% (n=150)
Append64Bytes/count=20-4        6.000 ± 0%   1.000 ± 0%  -83.33% (n=150)
Append64Bytes/count=50-4        7.000 ± 0%   1.000 ± 0%  -85.71% (n=150)
Append64Bytes/count=100-4       8.000 ± 0%   1.000 ± 0%  -87.50% (n=150)
Append64Bytes/count=200-4       9.000 ± 0%   1.000 ± 0%  -88.89% (n=150)
Append64Bytes/count=400-4      10.000 ± 0%   1.000 ± 0%  -90.00% (n=150)
geomean                             4.331        1.000       -76.91%

The second benchmark is similar, but instead uses an 8-byte integer
for the slice element. The first 4 appends in the loop never call into
the runtime thanks to the excellent CL 664299 introduced by Keith in
Go 1.25 that allows some <= 32 byte dynamically-sized slices to be on
the stack, so this CL is neutral for <= 32 bytes. Once the 5th append
occurs at count=5, a grow happens via the runtime and heap allocates
as normal, but freegc does not yet have anything to free, so we see
a small ~1.4ns penalty reported there. But once the second growth
happens, the older heap memory is now automatically freed by freegc,
so we start to see some benefit in memory reductions and speed
improvements, starting at a tiny speed improvement (close to a wash,
or maybe noise) by the second growth before count=10, and building up to
~2x faster with ~68% fewer allocated bytes reported.

goos: linux
goarch: amd64
pkg: runtime
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                       │ old-1bb1f2bf0c │        freegc-8ba7421-ps16        │
                       │     sec/op     │   sec/op     vs base              │
AppendInt/count=1-4         2.978n ± 0%   2.969n ± 0%   -0.30% (p=0.000 n=150)
AppendInt/count=4-4         4.292n ± 3%   4.163n ± 3%        ~ (p=0.528 n=150)
AppendInt/count=5-4         33.50n ± 0%   34.93n ± 0%   +4.25% (p=0.000 n=150)
AppendInt/count=10-4        76.21n ± 1%   75.67n ± 0%   -0.72% (p=0.000 n=150)
AppendInt/count=20-4        150.6n ± 1%   133.0n ± 0%  -11.65% (n=150)
AppendInt/count=50-4        284.1n ± 1%   225.6n ± 0%  -20.59% (n=150)
AppendInt/count=100-4       544.2n ± 1%   392.4n ± 1%  -27.89% (n=150)
AppendInt/count=200-4      1051.5n ± 1%   702.3n ± 0%  -33.21% (n=150)
AppendInt/count=400-4       2.041µ ± 1%   1.312µ ± 1%  -35.70% (n=150)
AppendInt/count=1000-4      5.224µ ± 2%   2.851µ ± 1%  -45.43% (n=150)
AppendInt/count=2000-4     11.770µ ± 1%   6.010µ ± 1%  -48.94% (n=150)
AppendInt/count=3000-4     17.747µ ± 2%   8.264µ ± 1%  -53.44% (n=150)
geomean                           331.8n        246.4n       -25.72%

                       │ old-1bb1f2bf0c │         freegc-8ba7421-ps16          │
                       │      B/op      │     B/op      vs base                │
AppendInt/count=1-4        0.000 ± 0%       0.000 ± 0%        ~ (p=1.000 n=150)
AppendInt/count=4-4        0.000 ± 0%       0.000 ± 0%        ~ (p=1.000 n=150)
AppendInt/count=5-4        64.00 ± 0%       64.00 ± 0%        ~ (p=1.000 n=150)
AppendInt/count=10-4       192.0 ± 0%       128.0 ± 0%  -33.33% (n=150)
AppendInt/count=20-4       448.0 ± 0%       256.0 ± 0%  -42.86% (n=150)
AppendInt/count=50-4       960.0 ± 0%       512.0 ± 0%  -46.67% (n=150)
AppendInt/count=100-4    1.938Ki ± 0%     1.000Ki ± 0%  -48.39% (n=150)
AppendInt/count=200-4    3.938Ki ± 0%     2.001Ki ± 0%  -49.18% (n=150)
AppendInt/count=400-4    7.938Ki ± 0%     4.005Ki ± 0%  -49.54% (n=150)
AppendInt/count=1000-4   24.56Ki ± 0%     10.05Ki ± 0%  -59.07% (n=150)
AppendInt/count=2000-4   58.56Ki ± 0%     20.31Ki ± 0%  -65.32% (n=150)
AppendInt/count=3000-4   85.19Ki ± 0%     27.30Ki ± 0%  -67.95% (n=150)
geomean                                     ²                 -42.81%

                       │ old-1bb1f2bf0c │        freegc-8ba7421-ps16         │
                       │   allocs/op    │ allocs/op   vs base                │
AppendInt/count=1-4        0.000 ± 0%     0.000 ± 0%        ~ (p=1.000 n=150)
AppendInt/count=4-4        0.000 ± 0%     0.000 ± 0%        ~ (p=1.000 n=150)
AppendInt/count=5-4        1.000 ± 0%     1.000 ± 0%        ~ (p=1.000 n=150)
AppendInt/count=10-4       2.000 ± 0%     1.000 ± 0%  -50.00% (n=150)
AppendInt/count=20-4       3.000 ± 0%     1.000 ± 0%  -66.67% (n=150)
AppendInt/count=50-4       4.000 ± 0%     1.000 ± 0%  -75.00% (n=150)
AppendInt/count=100-4      5.000 ± 0%     1.000 ± 0%  -80.00% (n=150)
AppendInt/count=200-4      6.000 ± 0%     1.000 ± 0%  -83.33% (n=150)
AppendInt/count=400-4      7.000 ± 0%     1.000 ± 0%  -85.71% (n=150)
AppendInt/count=1000-4     9.000 ± 0%     1.000 ± 0%  -88.89% (n=150)
AppendInt/count=2000-4    11.000 ± 0%     1.000 ± 0%  -90.91% (n=150)
AppendInt/count=3000-4    12.000 ± 0%     1.000 ± 0%  -91.67% (n=150)
geomean                                     ²               -72.76%                 ²

Of course, these are just microbenchmarks, but likely indicate
there are some opportunities here.

The immediately following CL 712422 tackles inlining and is able to get
runtime.freegc working automatically with iterators such as used by
slices.Collect, which becomes able to automatically free the
intermediate memory from its repeated appends (which earlier
in this work required a temporary hand edit to the slices package).

For now, we only use the NoAlias version for element types without
pointers while waiting on additional runtime support in CL 698515.

Updates #74299

Change-Id: I1b9d286aa97c170dcc2e203ec0f8ca72d84e8221
Reviewed-on: https://go-review.googlesource.com/c/go/+/710015
Reviewed-by: Keith Randall <khr@google.com>
Auto-Submit: Keith Randall <khr@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Keith Randall <khr@golang.org>

2 files changed, 75 insertions, 0 deletions

diff --git a/src/runtime/slice.go b/src/runtime/slice.go
index a9e8fc1610..2a44297754 100644
--- a/src/runtime/slice.go
+++ b/src/runtime/slice.go
@@ -7,6 +7,7 @@ package runtime
 import (
 	"internal/abi"
 	"internal/goarch"
+	"internal/goexperiment"
 	"internal/runtime/math"
 	"internal/runtime/sys"
 	"unsafe"
@@ -285,6 +286,42 @@ func growslice(oldPtr unsafe.Pointer, newLen, oldCap, num int, et *_type) slice
 	return slice{p, newLen, newcap}
 }
 
+// growsliceNoAlias is like growslice but only for the case where
+// we know that oldPtr is not aliased.
+//
+// In other words, the caller must know that there are no other references
+// to the backing memory of the slice being grown aside from the slice header
+// that will be updated with new backing memory when growsliceNoAlias
+// returns, and therefore oldPtr must be the only pointer to its referent
+// aside from the slice header updated by the returned slice.
+//
+// In addition, oldPtr must point to the start of the allocation and match
+// the pointer that was returned by mallocgc. In particular, oldPtr must not
+// be an interior pointer, such as after a reslice.
+//
+// See freegc for details.
+func growsliceNoAlias(oldPtr unsafe.Pointer, newLen, oldCap, num int, et *_type) slice {
+	s := growslice(oldPtr, newLen, oldCap, num, et)
+	if goexperiment.RuntimeFreegc && oldPtr != nil && oldPtr != s.array {
+		if gp := getg(); uintptr(oldPtr) < gp.stack.lo || gp.stack.hi <= uintptr(oldPtr) {
+			// oldPtr does not point into the current stack, and it is not
+			// the data pointer for s after the grow, so attempt to free it.
+			// (Note that freegc also verifies that oldPtr does not point into our stack,
+			// but checking here first is slightly cheaper for the case when
+			// oldPtr is on the stack and freegc would be a no-op.)
+			//
+			// TODO(thepudds): it may be that oldPtr==s.array only when elemsize==0,
+			// so perhaps we could prohibit growsliceNoAlias being called in that case
+			// and eliminate that check here, or alternatively, we could lean into
+			// freegc being a no-op for zero-sized allocations (that is, no check of
+			// oldPtr != s.array here and just let freegc return quickly).
+			noscan := !et.Pointers()
+			freegc(oldPtr, uintptr(oldCap)*et.Size_, noscan)
+		}
+	}
+	return s
+}
+
 // nextslicecap computes the next appropriate slice length.
 func nextslicecap(newLen, oldCap int) int {
 	newcap := oldCap
@@ -503,3 +540,22 @@ func growsliceBuf(oldPtr unsafe.Pointer, newLen, oldCap, num int, et *_type, buf
 
 	return slice{bufPtr, newLen, newCap}
 }
+
+// growsliceBufNoAlias is a combination of growsliceBuf and growsliceNoAlias.
+// bufPtr must be on the stack.
+func growsliceBufNoAlias(oldPtr unsafe.Pointer, newLen, oldCap, num int, et *_type, bufPtr unsafe.Pointer, bufLen int) slice {
+	s := growsliceBuf(oldPtr, newLen, oldCap, num, et, bufPtr, bufLen)
+	if goexperiment.RuntimeFreegc && oldPtr != bufPtr && oldPtr != nil && oldPtr != s.array {
+		// oldPtr is not bufPtr (the stack buffer) and it is not
+		// the data pointer for s after the grow, so attempt to free it.
+		// (Note that freegc does a broader check that oldPtr does not point into our stack,
+		// but checking here first is slightly cheaper for a common case when oldPtr is bufPtr
+		// and freegc would be a no-op.)
+		//
+		// TODO(thepudds): see related TODO in growsliceNoAlias about possibly eliminating
+		// the oldPtr != s.array check.
+		noscan := !et.Pointers()
+		freegc(oldPtr, uintptr(oldCap)*et.Size_, noscan)
+	}
+	return s
+}
diff --git a/src/runtime/slice_test.go b/src/runtime/slice_test.go
index 376b4a58f2..4779459c48 100644
--- a/src/runtime/slice_test.go
+++ b/src/runtime/slice_test.go
@@ -7,6 +7,7 @@ package runtime_test
 import (
 	"fmt"
 	"internal/asan"
+	"internal/goexperiment"
 	"internal/msan"
 	"internal/race"
 	"internal/testenv"
@@ -541,6 +542,18 @@ func TestAppendByteInLoop(t *testing.T) {
 		n := test[0]
 		want := test[1]
 		wantCap := test[2]
+
+		if goexperiment.RuntimeFreegc && n > 64 {
+			// Only 1 allocation is expected to be reported.
+			//
+			// TODO(thepudds): consider a test export or similar that lets us more directly see
+			// the count of freed objects, reused objects, and allocated objects. This could
+			// be in advance of any future runtime/metrics or user-visible API, or perhaps
+			// we could introduce the concept of build tag or debug flag controlled runtime/metrics
+			// targeting people working on the runtime and compiler (but not formally supported).
+			want = 1
+		}
+
 		var r []byte
 		got := testing.AllocsPerRun(10, func() {
 			r = byteSlice(n)
@@ -659,6 +672,12 @@ func TestAppendByteCapInLoop(t *testing.T) {
 		n := test[0]
 		want := test[1]
 		wantCap := test[2]
+
+		if goexperiment.RuntimeFreegc && n > 64 {
+			// Only 1 allocation is expected to be reported.
+			want = 1
+		}
+
 		var r []byte
 		got := testing.AllocsPerRun(10, func() {
 			r, _ = byteCapSlice(n)