runtime: use RDTSCP for instruction stream serialized read of TSC

To measure all instructions having been completed before reading
the time stamp counter with RDTSC an instruction sequence that
has instruction stream serializing properties which guarantee
waiting until all previous instructions have been executed is
needed. This does not necessary mean to wait for all stores to
be globally visible.

This CL aims to remove vendor specific logic for determining the
instruction sequence with CPU feature flag checks that are
CPU vendor independent.

For intel LFENCE has the wanted properties at least
since it was introduced together with SSE2 support.

On AMD instruction stream serializing LFENCE is supported by setting
an MSR C001_1029[1]=1 on AMD family 10h/12h/14h/15h/16h/17h processors.
AMD family 0Fh/11h processors support LFENCE as serializing always.
AMD plans support for this MSR and access to this bit for all future processors.
Source: https://developer.amd.com/wp-content/resources/Managing-Speculation-on-AMD-Processors.pdf

Reading the MSR to determine LFENCE properties is not always possible
or reliable (hypervisors). The Linux kernel is relying on serializing
LFENCE on AMD CPUs since a commit in July 2019: https://lkml.org/lkml/2019/7/22/295
and the MSR C001_1029 to enable serialization has been set by default
with the Spectre v1 mitigations.

Using an MFENCE on AMD is waiting on previous instructions having been executed
but in addition also flushes store buffers.

To align the serialization properties without runtime detection
of CPU manufacturers we can use the newer RDTSCP instruction which
waits until all previous instructions have been executed.

RDTSCP is available on Intel since around 2008 and on AMD CPUs since
around 2006. Support for RDTSCP can be checked independently
of manufacturer by checking CPUID bits.

Using RDTSCP is the default in Linux to read TSC in program order
when the instruction is available.
https://github.com/torvalds/linux/blob/e22ce8eb631bdc47a4a4ea7ecf4e4ba499db4f93/arch/x86/include/asm/msr.h#L231

Change-Id: Ifa841843b9abb2816f8f0754a163ebf01385306d
Reviewed-on: https://go-review.googlesource.com/c/go/+/344429
Reviewed-by: Keith Randall <khr@golang.org>
Trust: Martin Möhrmann <martin@golang.org>
Run-TryBot: Martin Möhrmann <martin@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>

2 files changed, 15 insertions, 0 deletions

diff --git a/src/internal/cpu/cpu.go b/src/internal/cpu/cpu.go
index dab5d068ef..a87d8a2314 100644
--- a/src/internal/cpu/cpu.go
+++ b/src/internal/cpu/cpu.go
@@ -36,6 +36,7 @@ var X86 struct {
 	HasOSXSAVE   bool
 	HasPCLMULQDQ bool
 	HasPOPCNT    bool
+	HasRDTSCP    bool
 	HasSSE2      bool
 	HasSSE3      bool
 	HasSSSE3     bool
diff --git a/src/internal/cpu/cpu_x86.go b/src/internal/cpu/cpu_x86.go
index fd1217a05d..a3f1fb809a 100644
--- a/src/internal/cpu/cpu_x86.go
+++ b/src/internal/cpu/cpu_x86.go
@@ -37,6 +37,9 @@ const (
 	cpuid_BMI2 = 1 << 8
 	cpuid_ERMS = 1 << 9
 	cpuid_ADX  = 1 << 19
+
+	// edx bits for CPUID 0x80000001
+	cpuid_RDTSCP = 1 << 27
 )
 
 var maxExtendedFunctionInformation uint32
@@ -53,6 +56,7 @@ func doinit() {
 		{Name: "fma", Feature: &X86.HasFMA},
 		{Name: "pclmulqdq", Feature: &X86.HasPCLMULQDQ},
 		{Name: "popcnt", Feature: &X86.HasPOPCNT},
+		{Name: "rdtscp", Feature: &X86.HasRDTSCP},
 		{Name: "sse3", Feature: &X86.HasSSE3},
 		{Name: "sse41", Feature: &X86.HasSSE41},
 		{Name: "sse42", Feature: &X86.HasSSE42},
@@ -112,6 +116,16 @@ func doinit() {
 	X86.HasBMI2 = isSet(ebx7, cpuid_BMI2)
 	X86.HasERMS = isSet(ebx7, cpuid_ERMS)
 	X86.HasADX = isSet(ebx7, cpuid_ADX)
+
+	var maxExtendedInformation uint32
+	maxExtendedInformation, _, _, _ = cpuid(0x80000000, 0)
+
+	if maxExtendedInformation < 0x80000001 {
+		return
+	}
+
+	_, _, _, edxExt1 := cpuid(0x80000001, 0)
+	X86.HasRDTSCP = isSet(edxExt1, cpuid_RDTSCP)
 }
 
 func isSet(hwc uint32, value uint32) bool {