[dev.fuzz] testing: make F.Fuzz more similar to T.Run

This change rewrites much of the glue code in testing/fuzz.go to work
more analogously to T.Run. This results in improved behavior:

* If a fuzz target returns without calling F.Skip, F.Fail, or F.Fuzz,
  'go test' will report an error and exit non-zero.
* Functions registered with F.Cleanup are called.
* The user can re-run individual inputs using -run=FuzzTarget/name
  where name is the base name of the seed corpus file. We now print
  the 'go test' command after a crash.

This change doesn't correctly handle T.Parallel calls yet, but it
should be easier to do that in the future.

Highlighted parts of this change:

* Instead of creating one F for all targets, create an F for each
  target. F (actually common) holds the status, output, and cleanup
  function list for each target, so it's important to keep them
  separate.
* Run each target in its own goroutine via fRunner. fRunner is
  analogous to tRunner. It runs cleanups and catches inappropriate
  Goexits and panics.
* Run each input in its own goroutine via T.Run. This enables subtest
  filtering with -test.run and ensures functions registered with
  T.Cleanup (not F.Cleanup) are run at the appropriate time.

Change-Id: Iab1da14ead8bcb57746f8a76f4aebc625baa5792
Reviewed-on: https://go-review.googlesource.com/c/go/+/290693
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>

1 files changed, 25 insertions, 1 deletions

diff --git a/src/internal/fuzz/fuzz.go b/src/internal/fuzz/fuzz.go
index 451731ba93..3b2baaf3a5 100644
--- a/src/internal/fuzz/fuzz.go
+++ b/src/internal/fuzz/fuzz.go
@@ -10,6 +10,7 @@ package fuzz
 import (
 	"context"
 	"crypto/sha256"
+	"errors"
 	"fmt"
 	"io/ioutil"
 	"os"
@@ -143,7 +144,10 @@ func CoordinateFuzzing(ctx context.Context, parallel int, seed []CorpusEntry, co
 			// A worker found a crasher. Write it to testdata and return it.
 			fileName, err := writeToCorpus(crasher.Data, corpusDir)
 			if err == nil {
-				err = fmt.Errorf("    Crash written to %s\n%s", fileName, crasher.errMsg)
+				err = &crashError{
+					name: filepath.Base(fileName),
+					err:  errors.New(crasher.errMsg),
+				}
 			}
 			// TODO(jayconrod,katiehockman): if -keepfuzzing, report the error to
 			// the user and restart the crashed worker.
@@ -181,6 +185,26 @@ func CoordinateFuzzing(ctx context.Context, parallel int, seed []CorpusEntry, co
 	// write to cacheDir instead.
 }
 
+// crashError wraps a crasher written to the seed corpus. It saves the name
+// of the file where the input causing the crasher was saved. The testing
+// framework uses this to report a command to re-run that specific input.
+type crashError struct {
+	name string
+	err  error
+}
+
+func (e *crashError) Error() string {
+	return e.err.Error()
+}
+
+func (e *crashError) Unwrap() error {
+	return e.err
+}
+
+func (e *crashError) CrashName() string {
+	return e.name
+}
+
 type corpus struct {
 	entries []CorpusEntry
 }