gotestyourself / gotestyourself/gotestsum
Accept additional json TestEvents from a side channel
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 171
- PR merge metrics
- No merged PRs in 30d
Description
### Background
This proposal aims to address two problems:
1. log output from background goroutines can not be correctly attributed to the right test when `t.Parallel` is used. I no longer use `t.Parallel` so it's hard for me to verify this is still a problem (https://github.com/golang/go/commit/1c72ee7f13831b215b8744f6b35bc4fd53aba5e2 may have fixed this, to be released in Go1.20). There are test suites out there that use `t.Parallel` extensively, and when I worked on them, this was a big problem. I believe the Go team has said this is working as intended and there is no way to fix this in the stdlib (TODO: find the relevant issue on the Go issue tracker). That's only true because the test binary doesn't directly output JSON, but it seems like that is not going to change.
2. tracking the timing of `TestMain` (see https://github.com/gotestyourself/gotestsum/issues/220#issuecomment-1352688384)
### Proposal
Note this is a big feature, so it should not be implemented unless there is really no way to avoid it.
A test suite can't send `TestEvent` json to `gotestsum` directly, because any output from the test suite is wrapped by `test2json`. For `gotestsum` to receive custom events there needs to be a side channel, a different output stream (not stdout or stderr).
I'm not sure if `gotestsum` could use `ExtraFiles` on https://pkg.go.dev/os/exec#Cmd when running the tests. I suspect because `go test` is between `gotestsum` and the test binary that the file descriptors won't be passed along. So most likely we'd have to use a file or pipe as a side channel for these events. A regular file is probably easier and more portable.
Essentially `gotestsum` would have a flag to specify a filename for side channel events, and would run `go test` with an extra flag to specify that filename:
```
gotestsum --extra-events-file=/tmp/testevents.log ./...
```
would run
```
go test ./... --extra-events-file=/tmp/testevents.log
```
The test suite would have to get the value from `--extra-events-file`, open that file, and write the newline delimited JSON events to it, as described here: https://pkg.go.dev/cmd/test2json#hdr-Output_Format
`gotestsum` would open that file as another source of events in https://github.com/gotestyourself/gotestsum/blob/10c2a47f9b99c8b49cbea60d12a6c76af0678d50/testjson/execution.go#L648-L654 and merge those into the `Execution`.
`gotestsum` not only needs the new flag, but `testjson.Execution` would also need to use locks to synchronize writes (or maybe build two separate `Execution` and merge them afterward?). And the test suite would also have to do some work to write those events to a file.
Maybe there's an easier way to create a side channel of events?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with testjson/execution.go around the event input path mentioned in the issue, then trace how gotestsum invokes go test and how test2json events enter Execution. Review the os/exec Cmd ExtraFiles limitation described here before choosing a side channel. Done would include a defined extra-events-file flow that reads newline-delimited JSON events and merges them safely into Execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100