gotestyourself / gotestyourself/gotestsum
Successful benchmark run is marked as failed
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 171
- PR merge metrics
- No merged PRs in 30d
Description
Hey all,
we observed gotestsum fail our benchmarks despite the benchmarks running fine. I tried to reproduce the issue and this was the smallest setup I could reproduce the issue with (using the latest gotestsum release v1.10.0):
Given a simple benchmark
```
import "testing"
func BenchmarkFuu(b *testing.B) {
l := 0
for i := 0; i < b.N; i++ {
l++
}
}
```
when run as e.g.
```
gotestsum --format standard-verbose --junitfile junit-results.xml --rerun-fails --rerun-fails-max-failures 10 --packages=. -- --bench=.
goos: darwin
goarch: arm64
=== RUN BenchmarkFuu
BenchmarkFuu
BenchmarkFuu-8 1000000000 0.3320 ns/op
PASS
ok ... 0.571s
=== Failed
=== FAIL: . BenchmarkFuu (unknown)
=== RUN BenchmarkFuu
BenchmarkFuu
BenchmarkFuu-8 1000000000 0.3320 ns/op
DONE 1 tests, 1 failure in 1.007s
```
it should not mark the test as failed. `go version` is `go version go1.20.4 darwin/arm64`.
I found https://github.com/gotestyourself/gotestsum/issues/62, is this related? Or do we use some incompatible configuration?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the issue with the provided BenchmarkFuu example, gotestsum v1.10.0, and the shown command and Go version. Compare the successful benchmark output with the generated failure summary; done means the run is reported as passing and the benchmark is not included as a failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100