matryer / matryer/is

Potential issue when running go test -json

Open
#47 4 comments 2 reactions 1 assignee View on GitHub

@flowchartsman is already working on this.

Since Mar 14, 2023.

Dominant language
Go
Stars
2k
Forks
67
PR merge metrics
No merged PRs in 30d

Description

Hey Mat, long time. Ran into a curious issue worth filing.

### Summary

Running `go test -json ...` and using this package reports incorrect test names in JSON events. This could very well be a bug in Go (test2json), but figured I'd start here because if I remove `is` and use regular std. lib primitives it reports the correct thing.

### Reproducible example

I noticed this happening in https://github.com/pressly/goose and put together a PR to showcase the problem.

In this commit I purposefully changed a test to trigger a failure. Note the name of the test is `TestNotAllowMissing`.

https://github.com/pressly/goose/pull/353/commits/9b7c7321bdbabb8307e71a40e0601d86b222d6ce#diff-080f0d7ba408eaf6181e1f61388995ebcd57fdd1468ffafffbf789176531418fR30

When I run `go test -v -json -count=1 ./tests/e2e -dialect=postgres` I happily get JSON output. But, the test name in the JSON event doesn't line up with the Output.

I was expecting Output that `is` prints to be associated with test named `TestNotAllowMissing`, but instead it was associated with another test named `TestMigrateFull`. (the name is non-deterministic, it changes between runs).

The raw output can be [found here ](https://github.com/pressly/goose/runs/6490014124?check_suite_focus=true#step:5:158). But I've copied and trimmed the relevant bits:

```
{"Test":"TestMigrateFull","Output":"\t\u001b[90mallow_missing_test.go:30: \u001b[39m7 != 8\n"}
{"Test":"TestNotAllowMissing","Output":"--- FAIL: TestNotAllowMissing (3.73s)\n"}
```

The first JSON event refers to `allow_missing_test.go:30`, in the code base the [call site on line 30](https://github.com/pressly/goose/blob/b44efc3a961c5c8be1876ba296f0ea6c7ef8cfdb/tests/e2e/allow_missing_test.go#L12-L30) is located within the test named: `TestNotAllowMissing`, but the test name is `TestMigrateFull`.

The second JSON event correctly outputs the failure under the test named `TestNotAllowMissing`.

This could very well be a bug in how go test prints JSON events, but I suspect it may be something within this library. I say this because if I remove `is` from that specific test (`TestNotAllowMissing`) I see the expected output associated with the correct Test name:

Raw output can be [found here](https://github.com/pressly/goose/runs/6490280411?check_suite_focus=true#step:5:160)

```
{"Test":"TestNotAllowMissing","Output":" allow_missing_test.go:35: got 7: want: 8\n"}
{"Test":"TestNotAllowMissing","Output":"--- FAIL: TestNotAllowMissing (2.81s)\n"}
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.