Potential issue when running go test -json
@flowchartsman is already working on this.
Since Mar 14, 2023.
- Dominant language
- Go
- Stars
- 2k
- Forks
- 67
- PR merge metrics
- No merged PRs in 30d
Description
Hey Mat, long time. Ran into a curious issue worth filing.
### Summary
Running `go test -json ...` and using this package reports incorrect test names in JSON events. This could very well be a bug in Go (test2json), but figured I'd start here because if I remove `is` and use regular std. lib primitives it reports the correct thing.
### Reproducible example
I noticed this happening in https://github.com/pressly/goose and put together a PR to showcase the problem.
In this commit I purposefully changed a test to trigger a failure. Note the name of the test is `TestNotAllowMissing`.
https://github.com/pressly/goose/pull/353/commits/9b7c7321bdbabb8307e71a40e0601d86b222d6ce#diff-080f0d7ba408eaf6181e1f61388995ebcd57fdd1468ffafffbf789176531418fR30
When I run `go test -v -json -count=1 ./tests/e2e -dialect=postgres` I happily get JSON output. But, the test name in the JSON event doesn't line up with the Output.
I was expecting Output that `is` prints to be associated with test named `TestNotAllowMissing`, but instead it was associated with another test named `TestMigrateFull`. (the name is non-deterministic, it changes between runs).
The raw output can be [found here ](https://github.com/pressly/goose/runs/6490014124?check_suite_focus=true#step:5:158). But I've copied and trimmed the relevant bits:
```
{"Test":"TestMigrateFull","Output":"\t\u001b[90mallow_missing_test.go:30: \u001b[39m7 != 8\n"}
{"Test":"TestNotAllowMissing","Output":"--- FAIL: TestNotAllowMissing (3.73s)\n"}
```
The first JSON event refers to `allow_missing_test.go:30`, in the code base the [call site on line 30](https://github.com/pressly/goose/blob/b44efc3a961c5c8be1876ba296f0ea6c7ef8cfdb/tests/e2e/allow_missing_test.go#L12-L30) is located within the test named: `TestNotAllowMissing`, but the test name is `TestMigrateFull`.
The second JSON event correctly outputs the failure under the test named `TestNotAllowMissing`.
This could very well be a bug in how go test prints JSON events, but I suspect it may be something within this library. I say this because if I remove `is` from that specific test (`TestNotAllowMissing`) I see the expected output associated with the correct Test name:
Raw output can be [found here](https://github.com/pressly/goose/runs/6490280411?check_suite_focus=true#step:5:160)
```
{"Test":"TestNotAllowMissing","Output":" allow_missing_test.go:35: got 7: want: 8\n"}
{"Test":"TestNotAllowMissing","Output":"--- FAIL: TestNotAllowMissing (2.81s)\n"}
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.