crypto: fix fips140=only detection of SHA-3

Both fips140only and the service indicator checks in
crypto/internal/fips140/... expect to type assert to
crypto/internal/fips140/{sha256,sha512,sha3}.Digest.

However, crypto/sha3 returns a wrapper concrete type around sha3.Digest.

Add a new fips140hash.Unwrap function to turn the wrapper into the
underlying sha3.Digest, and use it consistently before calling into
fips140only or the FIPS 140-3 module.

In crypto/rsa, also made the fips140only checks apply consistently after
the Go+BoringCrypto shims, so we can instantiate the hash, and avoid
having to wrap the New function. Note that fips140=only is incompatible
with Go+BoringCrypto.

Fixes #70879

Change-Id: I6a6a4656ec55c3e13f6cbfadb9cf89c0f9183bdc
Reviewed-on: https://go-review.googlesource.com/c/go/+/640855
Auto-Submit: Filippo Valsorda <filippo@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

1 files changed, 34 insertions, 0 deletions

diff --git a/src/crypto/internal/fips140hash/hash.go b/src/crypto/internal/fips140hash/hash.go
new file mode 100644
index 0000000000..6d67ee8b34
--- /dev/null
+++ b/src/crypto/internal/fips140hash/hash.go
@@ -0,0 +1,34 @@
+// Copyright 2024 The Go Authors. All rights reserved.
+// Use of this source code is governed by a BSD-style
+// license that can be found in the LICENSE file.
+
+package fips140hash
+
+import (
+	fsha3 "crypto/internal/fips140/sha3"
+	"crypto/sha3"
+	"hash"
+	_ "unsafe"
+)
+
+//go:linkname sha3Unwrap
+func sha3Unwrap(*sha3.SHA3) *fsha3.Digest
+
+// Unwrap returns h, or a crypto/internal/fips140 inner implementation of h.
+//
+// The return value can be type asserted to one of
+// [crypto/internal/fips140/sha256.Digest],
+// [crypto/internal/fips140/sha512.Digest], or
+// [crypto/internal/fips140/sha3.Digest] if it is a FIPS 140-3 approved hash.
+func Unwrap(h hash.Hash) hash.Hash {
+	if sha3, ok := h.(*sha3.SHA3); ok {
+		return sha3Unwrap(sha3)
+	}
+	return h
+}
+
+// UnwrapNew returns a function that calls newHash and applies [Unwrap] to the
+// return value.
+func UnwrapNew[Hash hash.Hash](newHash func() Hash) func() hash.Hash {
+	return func() hash.Hash { return Unwrap(newHash()) }
+}