gotestyourself / gotestyourself/gotestsum

Successful benchmark run is marked as failed

Open
#332 2 comments 0 reactions 0 assignees View on GitHub
test2json-bug
Dominant language
Go
Stars
2.7k
Forks
171
PR merge metrics
No merged PRs in 30d

Description

Hey all,

we observed gotestsum fail our benchmarks despite the benchmarks running fine. I tried to reproduce the issue and this was the smallest setup I could reproduce the issue with (using the latest gotestsum release v1.10.0):

Given a simple benchmark
```
import "testing"

func BenchmarkFuu(b *testing.B) {
l := 0
for i := 0; i < b.N; i++ {
l++
}
}
```
when run as e.g.
```
gotestsum --format standard-verbose --junitfile junit-results.xml --rerun-fails --rerun-fails-max-failures 10 --packages=. -- --bench=.

goos: darwin
goarch: arm64
=== RUN BenchmarkFuu
BenchmarkFuu
BenchmarkFuu-8 1000000000 0.3320 ns/op
PASS
ok ... 0.571s

=== Failed
=== FAIL: . BenchmarkFuu (unknown)
=== RUN BenchmarkFuu
BenchmarkFuu
BenchmarkFuu-8 1000000000 0.3320 ns/op

DONE 1 tests, 1 failure in 1.007s
```
it should not mark the test as failed. `go version` is `go version go1.20.4 darwin/arm64`.

I found https://github.com/gotestyourself/gotestsum/issues/62, is this related? Or do we use some incompatible configuration?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the provided BenchmarkFuu example, gotestsum v1.10.0, and the shown command and Go version. Compare the successful benchmark output with the generated failure summary; done means the run is reported as passing and the benchmark is not included as a failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.