| Age | Commit message (Collapse) | Author |
|---|
|
Adds a mechanism for generating test2json test artifacts from and validating them against a real test. If a .test file has a corresponding .src file, TestGolden will now treat the .src file as a script test, executing it and verifying that the output matches the contents of the .test file. Running TestGolden with the -update flag will also regenerate .test files if they have a corresponding .src file.
Capturing the output of the script test in this way required making minor changes to cmd/internal/script/scripttest.
This was motivated by CL 601535 (golang/go#62728). Specifically, testing that CL required adding src/cmd/internal/test2json/testdata/frameescape.test which has a multitude of non-printing characters and thus must be generated by executing `go test`. Using a script test to generate the test file is more reliable than doing it by hand.
Change-Id: I60456700e37e21a42d0514be2ce86dc6df2bccb0
Reviewed-on: https://go-review.googlesource.com/c/go/+/628615
Reviewed-by: Michael Matloob <matloob@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Matloob <matloob@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Damien Neil <dneil@google.com>
|
|
Provides a way to disambiguate output produced by (*testing.T).Error{,f}
and (*testing.T).Fatal{,f} from other test logging. This allows test
tooling such as CI systems to identify which part of the output is most
pertinent for constructing summaries of test failures.
This is achieved by adding an OutputType field to output events. The
output type for an error is "error" for the first line and
"error-continue" for subsequentlines. The output type for framing is
"frame". This is achieved by bracketing error output with ^O and ^N,
escaped with ^[.
Fixes golang/go#62728.
Change-Id: Ib09c18ed5f729e1ae6d335cd1ec7d818c71532e0
Reviewed-on: https://go-review.googlesource.com/c/go/+/601535
Reviewed-by: Michael Matloob <matloob@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Damien Neil <dneil@google.com>
Auto-Submit: Damien Neil <dneil@google.com>
|
|
Uses `^[` to escape the framing marker `^V` used to delimit test output. A test
that itself executes a go test binary, or otherwise emits that control
character, previously would corrupt the test2json parse of the enclosing run.
Updates #62728.
Change-Id: I0e8790a05fd7af469cd7ee2e8ccc13786cc372dc
Reviewed-on: https://go-review.googlesource.com/c/go/+/751940
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Matloob <matloob@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
|
|
Add TB.ArtifactDir, which returns a directory for a test to store
output files in. Add a -artifacts testflag which enables persistent
storage of artifacts in the output directory (-outputdir, or the
current directory by default).
Fixes #71287
Change-Id: I5f6515a6cd6c103f88588f4c033d5ea11ffd0c3c
Reviewed-on: https://go-review.googlesource.com/c/go/+/696399
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
|
|
Add a new Attr method to testing.TB that emits a test attribute.
An attribute is an arbitrary key/value pair.
Fixes #43936
Change-Id: I7ef299efae41f2cf39f2dc61ad4cdd4c3975cdb6
Reviewed-on: https://go-review.googlesource.com/c/go/+/662437
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
Auto-Submit: Damien Neil <dneil@google.com>
|
|
Currently, if a test or imported package fails to build during "go
test -json", the build error text will be interleaved with the JSON
output of tests. Furthermore, there’s currently no way to reliably
associate a build error with the test package or packages it affected.
This creates unnecessary friction and complexity in tools that consume
the "go test -json" output.
This CL makes "go test -json" enable JSON reporting of build errors.
It also adds a "FailedBuild" field to the "fail" TestEvent, which
gives the package ID of the package that failed to build and caused
the test to fail.
Using this, CI systems should be able to consume the entire output
stream from "go test -json" in a structured way and easily associate
build failures with test failures during reporting.
Fixes #62067.
Updates #35169.
Updates #37486.
Change-Id: I49091dcc7aa52db01fc9fa6042771633e97b8407
Reviewed-on: https://go-review.googlesource.com/c/go/+/536399
Reviewed-by: Russ Cox <rsc@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
|
|
Add a new "Action":"start" test2json event to mark the
start of the test binary execution. This adds useful information
to the JSON traces, and it also lets programs watching test
execution see the order in which the tests are being run,
because we arrange for the starts to happen sequentially.
Change-Id: I9fc865a486a55a7e9315f8686f59a2aa06455884
Reviewed-on: https://go-review.googlesource.com/c/go/+/448357
Run-TryBot: Russ Cox <rsc@golang.org>
Auto-Submit: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
|
|
Test2json is parsing the output stream from the test, which includes
package testing's own framing lines intermingled with other output,
in particular any output printed via fmt.Printf, println, and so on.
We have had recurring problems with unexpected partial output lines
causing a framing line to be missed.
A recent talk at GopherCon gave an example of an integration test
involving Docker that happened to print \r-terminated lines instead
of \n-terminated lines in some configurations, which in turn broke
test2json badly. (https://www.gophercon.com/agenda/session/944259)
There are also a variety of open reported issues with similar problems,
which this CL also addresses. The general approach is to add a new
testing flag -test.v=json that means to print additional output to help
test2json. And then test2json takes advantage of that output.
Among the fixes:
- Identify testing framing more reliably, using ^V
(#23036, #26325, #43683, GopherCon talk)
- Test that output with \r\n endings is handled correctly
(#43683, #34286)
- Use === RUN in fuzz tests (#52636, #48132)
- Add === RUN lines to note benchmark starts (#27764, #49505)
- Print subtest --- PASS/FAIL lines as they happen (#29811)
- Add === NAME lines to emit more test change events,
such as when a subtest stops and the parent continues running.
- Fix event shown in overall test failure (#27568)
- Avoid interleaving of writes to os.Stdout and os.Stderr (#33419)
Fixes #23036.
Fixes #26325.
Fixes #27568.
Fixes #27764.
Fixes #29811.
Fixes #33419.
Fixes #34286.
Fixes #43683.
Fixes #49505.
Fixes #52636.
Change-Id: Id4207b746a20693f92e52d68c6e4a7f8c41cc7c6
Reviewed-on: https://go-review.googlesource.com/c/go/+/443596
Auto-Submit: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
|
|
Change-Id: Ifa78c98bf919ea62136f19b2bad0a8ee33afc646
Reviewed-on: https://go-review.googlesource.com/c/go/+/435695
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Robert Griesemer <gri@google.com>
Run-TryBot: xie cui <523516579@qq.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
|
|
For #45557
Change-Id: I56824135d86452603dd4ed4bab0e24c201bb0683
Reviewed-on: https://go-review.googlesource.com/c/go/+/426257
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Run-TryBot: Andy Pan <panjf2000@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
|
|
[This CL is part of a sequence implementing the proposal #51082.
The design doc is at https://go.dev/s/godocfmt-design.]
Run the updated gofmt, which reformats doc comments,
on the main repository. Vendored files are excluded.
For #51082.
Change-Id: I7332f099b60f716295fb34719c98c04eb1a85407
Reviewed-on: https://go-review.googlesource.com/c/go/+/384268
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
|
And then revert the bootstrap cmd directories and certain testdata.
And adjust tests as needed.
Not reverting the changes in std that are bootstrapped,
because some of those changes would appear in API docs,
and we want to use any consistently.
Instead, rewrite 'any' to 'interface{}' in cmd/dist for those directories
when preparing the bootstrap copy.
A few files changed as a result of running gofmt -w
not because of interface{} -> any but because they
hadn't been updated for the new //go:build lines.
Fixes #49884.
Change-Id: Ie8045cba995f65bd79c694ec77a1b3d1fe01bb09
Reviewed-on: https://go-review.googlesource.com/c/go/+/368254
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
|
|
If we are parsing a test output, and the test does not end in the
usual PASS or FAIL line (say, because it panicked), then we need the
exit status of the test binary in order to determine whether the test
passed or failed. If we don't have that status available, we shouldn't
guess arbitrarily — instead, we should omit the final "pass" or "fail"
action entirely.
(In practice, we nearly always DO have the final status, such as when
running 'go test' or 'go tool test2json some.exe'.)
Fixes #40132
Change-Id: Iae482577361a6033395fe4a05d746b980e18c3de
Reviewed-on: https://go-review.googlesource.com/c/go/+/248624
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
|
|
When printing regular test output check the indentation of the output, and use
the report stack to find the appropriate test name for that output.
This change includes a whitespace change to some golden test files. The
indentation of tests was changed in CL 113177
from tabs to spaces. The golden files have been updated to match the new
output format. The tabs in the golden files cause problems because the indentation check
looks for 4 spaces.
Fixes #29755
Updates #25369
Change-Id: Iebab51816a9755168083a7a665b41497e9dfd85f
GitHub-Last-Rev: 898827f1a6a163fd81dc667f5d27fd4893260038
GitHub-Pull-Request: golang/go#34419
Reviewed-on: https://go-review.googlesource.com/c/go/+/196617
Reviewed-by: Bryan C. Mills <bcmills@google.com>
|
|
'go test -json' should report that a test failed if the test binary
did not exit normally with status 0. This covers panics, non-zero
exits, and abnormal terminations.
These tests don't print a final result when run with -test.v (which is
used by 'go test -json'). The final result should be "PASS" or "FAIL"
on a line by itself. 'go test' prints "FAIL" in this case, but
includes error information.
test2json was changed in CL 192104 to report that a test passed if it
does not report a final status. This caused 'go test -json' to report
that a test passed after a panic or non-zero exit.
With this change, test2json treats "FAIL" with error information the
same as "FAIL" on a line by itself. This is intended to be a minimal
fix for backporting, but it will likely be replaced by a complete
solution for #29062.
Fixes #37555
Updates #29062
Updates #31969
Change-Id: Icb67bcd36bed97e6a8d51f4d14bf71f73c83ac3d
Reviewed-on: https://go-review.googlesource.com/c/go/+/222243
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
|
|
Flush the output log up to the root when a test panics. Prior to
this change, only the current test's output log was flushed to its
parent, resulting in no output when a subtest panics.
For the following test function:
func Test(t *testing.T) {
for i, test := range []int{1, 0, 2} {
t.Run(fmt.Sprintf("%v/%v", i, test), func(t *testing.T) {
_ = 1 / test
})
}
}
Output before this change:
panic: runtime error: integer divide by zero [recovered]
panic: runtime error: integer divide by zero
(stack trace follows)
Output after this change:
--- FAIL: Test (0.00s)
--- FAIL: Test/1/0 (0.00s)
panic: runtime error: integer divide by zero [recovered]
(stack trace follows)
Fixes #32121
Change-Id: Ifee07ccc005f0493a902190a8be734943123b6b7
Reviewed-on: https://go-review.googlesource.com/c/go/+/179599
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
|
When a test has a TestMain that doesn't run any tests (doesn't invoke
m.Run), `go test` passes, but `go test -json` reports a "fail" event
though the exit code is still 0.
This CL fixes test2json to behave similarly to `go test` in such cases -
no output from the test is taken as "pass" by default, not as "fail".
Fixes #31969
Change-Id: I1829d40fc30dc2879e73974fac416f6a34212ccd
Reviewed-on: https://go-review.googlesource.com/c/go/+/192104
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
|
Original CL description:
When using test -cover or -coverprofile the output for "no test files"
is the same format as for "no tests to run".
Reverting because this CL changed cmd/go to build test binaries for
packages that have no tests, leading to extra work and confusion.
Updates #24570
Fixes #25789
Fixes #26157
Fixes #26242
Change-Id: Ibab1307d39dfaec0de9359d6d96706e3910c8efd
Reviewed-on: https://go-review.googlesource.com/122518
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
|
|
When using test -cover or -coverprofile the output for "no test files"
is the same format as for "no tests to run".
Fixes #24570
Change-Id: If05609411676d42d94c1feac4bc839974fae2cc1
Reviewed-on: https://go-review.googlesource.com/115095
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
|
The "updates" lines, such as RUN, do not contain a colon. However,
test2json looked for one anyway, meaning that it would be thrown off if
it encountered a line like:
=== RUN TestWithColons/[::1]
In that case, it must not use the first colon it encounters to separate
the action from the test name.
Fixes #23920.
Change-Id: I82eff23e24b83dae183c0cf9f85fc5f409f51c25
Reviewed-on: https://go-review.googlesource.com/98445
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
|
When benchmarks run, they print lines like:
BenchmarkGenericNoMatch-8 3000000 385 ns/op
The first field, padded by spaces and followed by a tab,
is printed when the benchmark begins running.
The rest of the line is printed when the benchmark ends.
Tools and people can watch the timing of these prints
to see which benchmark is running.
To allow tools consuming json output to continue to be
able to see which benchmark is running, this CL adds a
special case to the usual "line at a time" behavior to flush
the benchmark name if it is observed separately from the
rest of the line.
Fixes #23352.
Change-Id: I7b6410698d78034eec18745d7f57b7d8e9575dbb
Reviewed-on: https://go-review.googlesource.com/86695
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
|
If a benchmark calls b.Log without failing (without b.Error/b.Fatal/b.FailNow)
then that turns into output very much like a test passing,
except it says BENCH instead of PASS.
Benchmarks failing say FAIL just like tests failing.
Fixes #23346.
Change-Id: Ib188e695952da78057ab4a13f90d49937aa3c232
Reviewed-on: https://go-review.googlesource.com/86396
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
|
If test case framing appears in ordinary test output,
then test2json can get confused. If the fake framing is being
saved with t.Logf/t.Errorf/etc then we can already
distinguish it from real framing, and the code did.
It just forgot to write that framing as output (1-line fix).
If the fake framing is being generated by printing directly
to stdout/stderr, then test2json will simply get confused.
There's not a lot to do at that point (maybe it's even a feature).
Fixes #23036.
Change-Id: I29449c7ace304172b89d8babe23de507c0500455
Reviewed-on: https://go-review.googlesource.com/86238
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
|
1. Apply JSON conversion when -bench is in use.
2. Apply JSON conversion to "no test files" result.
3. Apply JSON conversion to test case-ending SKIP status.
Fixes #22769.
Fixes #22790.
Change-Id: I67ad656fc58bacae8c51d23b1e6d543cad190f08
Reviewed-on: https://go-review.googlesource.com/81535
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
|
|
Also add cmd/internal/test2json, the actual implementation,
which will be called directly from cmd/go in addition to being
a standalone command (like cmd/buildid and cmd/internal/buildid).
For #2981.
Change-Id: I244ce36d665f424bbf13f5ae00ece10b705d367d
Reviewed-on: https://go-review.googlesource.com/76872
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
|
