| Age | Commit message (Collapse) | Author |
|---|
|
This removes redundancy for the default value.
Fixes #46555
Change-Id: Ib62bd2d584ef82bef806d0fe2ce59957488e469e
Reviewed-on: https://go-review.googlesource.com/c/go/+/325070
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
|
|
When GODEBUG=fuzzdebug=1, log additional debug level information about
what the fuzzer is doing. This provides useful information for
investigating the operation and performance of the fuzzing engine, and
is necessary for profiling new fuzzing strategies.
Change-Id: Ic3e24e7a128781377e62785767a218811c3c2030
Reviewed-on: https://go-review.googlesource.com/c/go/+/324972
Trust: Roland Shoemaker <roland@golang.org>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Roland Shoemaker <roland@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
And only log the last panic, not all of them, during minimization.
This change makes the worker processes quiet, so now the only
process that logs anything is the coordinator. This hides all of
the panics caused during minimization of an input which causes
a panic.
This change also alters the usage of tRunner such that we now
recover from recoverable panics instead of terminating the
process. This results in larger stack traces, since we include
a bit more of the trace within testing. There is a TODO to see
if it's possible to slice the stack up so that it is somewhat
more informative.
Change-Id: Ic85eabd2e70b078412fbb88adf424a8da25af876
Reviewed-on: https://go-review.googlesource.com/c/go/+/321230
Trust: Roland Shoemaker <roland@golang.org>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Roland Shoemaker <roland@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
* Introduced -fuzzminimizetime flag to control the number of time or
the number of calls to spend minimizing. Defaults to 60s. Only works
for unrecoverable crashes for now.
* Moved the count (used by -fuzztime=1000x) into shared
memory. Calling workerClient.fuzz resets it, but it will remain
after the worker processes crashes. workerClient.minimize resets it
once before restarting the worker the first time, but the total
number of runs should still be limited during minimization, even
after multiple terminations and restarts.
* Renamed fuzzArgs.Count to Limit to avoid confusion.
* Several other small fixes and refactorings.
Change-Id: I03faa4c94405041f6dfe48568e5ead502f8dbbd2
Reviewed-on: https://go-review.googlesource.com/c/go/+/320171
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
When instrumented packages intersect with the packages used by the
testing or internal/fuzz packages the coverage counters become noisier,
as counters will be triggered by non-fuzzed harness code.
Ideally counters would be deterministic, as there are many advanced
fuzzing strategies that require mutating the input while maintaining
static coverage.
The simplest way to mitigate this noise is to capture the coverage
counters as closely as possible to the invocation of the fuzz target
in the testing package. In order to do this add a new function which
captures the current values of the counters, SnapshotCoverage. This
function copies the current counters into a static buffer,
coverageSnapshot, which workerServer.fuzz can then inspect when it
comes time to check if new coverage has been found.
This method is not foolproof. As the fuzz target is called in a
goroutine, harness code can still cause counters to be incremented
while the target is being executed. Despite this we do see
significant reduction in churn via this approach. For example,
running a basic target that causes strconv to be instrumented for
500,000 iterations causes ~800 unique sets of coverage counters,
whereas by capturing the counters closer to the target we get ~40
unique sets.
It may be possible to make counters completely deterministic, but
likely this would require rewriting testing/F.Fuzz to not use tRunner
in a goroutine, and instead use it in a blocking manner (which I
couldn't figure out an obvious way to do), or by doing something even
more complex.
Change-Id: I95c2f3b1d7089c3e6885fc7628a0d3a8ac1a99cf
Reviewed-on: https://go-review.googlesource.com/c/go/+/320329
Trust: Roland Shoemaker <roland@golang.org>
Trust: Katie Hockman <katie@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
Usage of f.testContext.match.fullName to generate the test name causes
unbounded memory growth, eventually causing the fuzzer to slow down
as memory pressure increases.
Each time fuzzFn is invoked it generates a unique string and stores it
in a map. With the fuzzer running at around 100k executions per second
this consumed around ~30GB of memory in a handful of minutes.
Instead just use the base name of the test for mutated inputs, a special
name for seeded inputs, and the filename for inputs from the input
corpus.
Change-Id: I083f47df7e82f0c6b0bda244f158233784a13029
Reviewed-on: https://go-review.googlesource.com/c/go/+/316030
Trust: Roland Shoemaker <roland@golang.org>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Roland Shoemaker <roland@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
While running the seed corpus, T.Parallel acts like it does in
subtests started with T.Run: it blocks until all other non-parallel
subtests have finished, then unblocks when the barrier chan is
closed. A semaphore (t.context.waitParallel) limits the number of
tests that run concurrently (determined by -test.parallel).
While fuzzing, T.Parallel has no effect, other than asserting that it
can't be called multiple times. We already run different inputs in
concurrent processes, but we can't run inputs concurrently in the same
process if we want to attribute crashes to specific inputs.
Change-Id: I2bac08e647e1d92ea410c83c3f3558a033fe3dd1
Reviewed-on: https://go-review.googlesource.com/c/go/+/300449
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
-fuzztime now works similarly to -benchtime: if it's given a string
with an "x" suffix (as opposed to "s" or some other unit of
duration), the fuzzing system will generate and run a maximum number
of values.
This CL also implements tracking and printing counts, since most of
the work was already done.
Change-Id: I013007984b5adfc1a751c379dc98c8d46b4a97e9
Reviewed-on: https://go-review.googlesource.com/c/go/+/306909
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
worker.runFuzzing now accepts a Context, used for cancellation instead
of doneC (which is removed). This is passed down through workerClient
RPC methods (ping, fuzz).
workerClient RPC methods now wrap the call method, which handles
marshaling and cancellation.
Both workerClient.call and workerServer.serve should return quickly
when their contexts are cancelled. Turns out, closing the pipe won't
actually unblock a read on all platforms. Instead, we were falling
back to SIGKILL in worker.stop, which works but takes longer than
necessary.
Also fixed missing newline in log message.
Change-Id: I7b5ae54d6eb9afd6361a07759f049f048952e0cc
Reviewed-on: https://go-review.googlesource.com/c/go/+/303429
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
Also improve the error messages for the use of
testing.F functions inside the Fuzz function.
Change-Id: I5fa48f8c7e0460a1da89a49a73e5af83c544e549
Reviewed-on: https://go-review.googlesource.com/c/go/+/298849
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
If a worker process encounters an error communicating with the
coordinator, or if the setup code reports an error with F.Fail
before calling F.Fuzz, exit with status 70. The coordinator will report
these errors and 'go test' will exit non-zero, but the coordinator
won't record a crasher since the problem is not in the code being
fuzzed.
The coordinator also detects unexpected terminations before the worker
calls F.Fuzz by sending a ping RPC. If the worker terminates before
responding to the ping RPC, the coordinator won't record a crasher.
Exit codes are chosen somewhat arbitrary, but in the Advanced Bash
Scripting Guide, 70 is "internal software error" which is applicable
here. 70 is also ASCII 'F'.
Change-Id: I1e676e39a7b07c5664efaaa3221d055f55240fff
Reviewed-on: https://go-review.googlesource.com/c/go/+/297033
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
* Run gofmt with go1.17 build constraint changes.
* Tighten regular expressions used in tests. "ok" got some false
positives with verbose output, so make sure it appears at the start
of a line.
* Return err in deps.RunFuzzWorker instead of nil.
* Call common.Helper from F methods. This prevents F methods from
appearing in stack traces.
Change-Id: I839c70ec8a9f313c1a4ea68e6bb34a4d801dd36f
Reviewed-on: https://go-review.googlesource.com/c/go/+/297032
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
This change makes several refactors to start supporting
structured fuzzing. The mutator can still only mutate
byte slices, and future changes will be made to support
mutating other types. However, it does now support
fuzzing more than one []byte.
This change also makes it so that corpus entries are
encoded in the new file format when being written to
testdata or GOCACHE. Any existing GOCACHE data should
be deleted from your local workstation to allow tests
to pass locally.
Change-Id: Iab8fe01a5dc870f0c53010b9d5b0b479bbdb310d
Reviewed-on: https://go-review.googlesource.com/c/go/+/293810
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
This change rewrites much of the glue code in testing/fuzz.go to work
more analogously to T.Run. This results in improved behavior:
* If a fuzz target returns without calling F.Skip, F.Fail, or F.Fuzz,
'go test' will report an error and exit non-zero.
* Functions registered with F.Cleanup are called.
* The user can re-run individual inputs using -run=FuzzTarget/name
where name is the base name of the seed corpus file. We now print
the 'go test' command after a crash.
This change doesn't correctly handle T.Parallel calls yet, but it
should be easier to do that in the future.
Highlighted parts of this change:
* Instead of creating one F for all targets, create an F for each
target. F (actually common) holds the status, output, and cleanup
function list for each target, so it's important to keep them
separate.
* Run each target in its own goroutine via fRunner. fRunner is
analogous to tRunner. It runs cleanups and catches inappropriate
Goexits and panics.
* Run each input in its own goroutine via T.Run. This enables subtest
filtering with -test.run and ensures functions registered with
T.Cleanup (not F.Cleanup) are run at the appropriate time.
Change-Id: Iab1da14ead8bcb57746f8a76f4aebc625baa5792
Reviewed-on: https://go-review.googlesource.com/c/go/+/290693
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
|
|
inFuzzFn is set when the fuzz function is called. While it's set,
F methods that have side effects like Skip and Fail may not be called.
Previously, (CL 259657) inFuzzFn was in common, and we checked it in
the common implementation of those methods. This causes problems in
CL 290693 for recursive methods like common.Fail. If T.Fail is
called by the fuzz function, it calls common.Fail on the parent F's
common. That should not panic.
Change-Id: I841b12f77d9c77f5021370d03313e71b4ef50102
Reviewed-on: https://go-review.googlesource.com/c/go/+/290811
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
RunFuzzWorker now accepts a fuzz.CorpusEntry instead of []byte. This
may help us pass structured data in the future.
Change-Id: Idf7754cb890b6a835d887032fd23ade4d0713bcf
Reviewed-on: https://go-review.googlesource.com/c/go/+/290692
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
CorpusEntry is now a struct type with Name and Data fields. In the
future, it may have more fields describing multiple values with
different types added with f.Add.
CorpusEntry must be the same type in testing and
internal/fuzz. However, we don't want to export it from testing, and
testing can't import internal/fuzz. We define it to be a type alias of
a struct type instead of a defined type. We need to define it to the
same thing in both places. We'll get a type error when building cmd/go
if there's a difference.
Change-Id: I9df6cd7aed67a6aa48b77ffb3a84bd302d2e5d94
Reviewed-on: https://go-review.googlesource.com/c/go/+/288534
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
|
|
fuzz.CoordinateFuzzing and RunFuzzWorker now accept a context.Context
parameter. They should terminate gracefully when the context is
cancelled. The worker should exit quickly without processing more
inputs. The coordinator should save interesting inputs to the cache.
The testing package can't import context directly, so it provides a
timeout argument to testdeps.CoordinateFuzzing instead. The testdeps
wrapper sets the timeout and installs an interrupt handler (for SIGINT
on POSIX and the equivalent on Windows) that cancels the context when
^C is pressed.
Note that on POSIX platforms, pressing ^C causes the shell to deliver
SIGINT to all processes in the active group: so 'go test', the
coordinator, and the workers should all react to that. On Windows,
pressing ^C only interrupts 'go test'. We may want to look at that
separately.
Change-Id: I924d3be2905f9685dae82ff3c047ca3d6b5e2357
Reviewed-on: https://go-review.googlesource.com/c/go/+/279487
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
|
|
'go test -fuzz' may now read and write interesting fuzzing values to
directories in $GOCACHE/fuzz. Files in this directory are named
$pkg/$test/$hash where $pkg is the package path containing the fuzz
target, $test is the target name, and $hash is the SHA-256 sum of the
data in the file.
Note that different versions of the same package or packages with the
same path from different modules may share the same directory.
Although files are written into a subdirectory of GOCACHE, they are
not removed automatically, nor are they removed by 'go clean -cache'.
Instead, they may be removed with 'go clean -fuzzcache'. We chose to
nest the fuzzing directory inside GOCACHE to avoid introducing a new
environment variable, since there's no real need for users to specify
a separate directory.
Change-Id: I2032cf8e6c92f715cf36a9fc6a550acf666d2382
Reviewed-on: https://go-review.googlesource.com/c/go/+/275534
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
|
|
Change-Id: Ie2a84c12f4991984974162e74f06cfd67e9bb4d7
Reviewed-on: https://go-review.googlesource.com/c/go/+/274855
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
The workers were printing PASS/FAIL logs and
various others things, when that should be
the sole responsibility of the coordinator
process, which will have the aggregated data.
Change-Id: I7ac9883db62f0fe79ba1799cb88773c542a2a948
Reviewed-on: https://go-review.googlesource.com/c/go/+/274652
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
The coordinator process creates a temporary file for each worker. Both
coordinator and worker map the file into memory and use it for input
values. Access is synchronized with RPC over pipes.
Change-Id: I43c10d7291a8760a616b472d11c017a3a7bb19cf
Reviewed-on: https://go-review.googlesource.com/c/go/+/263153
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
|
|
Deletes the exported testing.Fuzz function
which would run a standalone fuzz target.
Similar to RunFuzzing and RunFuzzTargets,
which were previously removed, this will
likely be too complex to support.
Moves the deferred Exit in f.Fuzz higher up
the function so it is always run.
Change-Id: I9ea6210dc30dee8c2a943bfb8077225c369cfb95
Reviewed-on: https://go-review.googlesource.com/c/go/+/263642
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
This change also includes a small cleanup of the run()
function and additional tests for error conditions
in fuzz targets.
Change-Id: I2b7722b25a0d071182a84f1dc4b92e82a7ea34d9
Reviewed-on: https://go-review.googlesource.com/c/go/+/256978
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
It is a legacy practice to expose these exported
testing functions, and is not needed for running
fuzz targets.
Change-Id: Ic300c9bfd15f4e71a1cea99f12c97d671a899f9b
Reviewed-on: https://go-review.googlesource.com/c/go/+/262258
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
Package fuzz provides common fuzzing functionality for tests built
with "go test" and for programs that use fuzzing functionality in the
testing package.
Change-Id: I3901c6a993a9adb8a93733ae1838b86dd78c7036
Reviewed-on: https://go-review.googlesource.com/c/go/+/259259
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
|
|
This change causes f.Fuzz to call runtime.GoExit
when it has finished running. This would mean that
any code after an f.Fuzz function within a fuzz
target would not be executed.
In the future, vet should fail if someone tries to
do this.
This change also adds the missing code that would
execute any cleanup functions added by f.Cleanup.
Change-Id: Ib4d1e6bcafbe189986d0667a1e87dabae67ee621
Reviewed-on: https://go-review.googlesource.com/c/go/+/260338
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
Change-Id: I8ee513b2b157e6033d4bc9607d0e65f42bd6801f
Reviewed-on: https://go-review.googlesource.com/c/go/+/259657
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
Change-Id: Ibd204a5d0596c4f8acf598289055c17a836d9023
Reviewed-on: https://go-review.googlesource.com/c/go/+/255957
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Katie Hockman <katie@golang.org>
Trust: Jay Conrod <jayconrod@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
Change-Id: I183693fec6a643b2f27cc379a422e2b42d8eca90
Reviewed-on: https://go-review.googlesource.com/c/go/+/255339
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
|
|
Change-Id: Idcf90c5acbf7dbba2ea01d21d893214a5c2028c2
Reviewed-on: https://go-review.googlesource.com/c/go/+/255517
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
Tests include:
- matching fuzz targets
- matching fuzz targets with -fuzz
- chatty tests with -v
- failing tests
- skipped tests
- passing tests
- panic in tests
Change-Id: I54e63c8891b45cfae7212924e067e790f25ab411
Reviewed-on: https://go-review.googlesource.com/c/go/+/254360
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
|
|
This change adds support for a -fuzz flag in the go command, and sets up
the groundwork for native fuzzing support. These functions are no-ops
for now, but will be built out and tested in future PRs.
Change-Id: I58e78eceada5799bcb73acc4ae5a20372badbf40
Reviewed-on: https://go-review.googlesource.com/c/go/+/251441
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
