gotestyourself / gotestyourself/gotestsum
Detect Flaky tests in an optimized way
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 171
- PR merge metrics
- No merged PRs in 30d
Description
Right now, I am trying to detect flaky tests, but I think looping over the test_results.json file will be inefficient since I would need to go through all packages. However, I realized that the output of the gotestsum command lists the different packages being tested, along with whether each test succeeded or failed. At the end of the output, there is a `=== Failed` section.
For example, the output in the case of the github-actions format might look like this:
```
::group::PASS cmd.TestStable (0.00s)
main_test.go:9: This is a stable test
::endgroup::
::group::FAIL cmd.Test_Flaky2 (0.00s)
main_test.go:21: Flaky test failed!
::endgroup::
FAIL Package cmd (297ms)
PASS mathutils.TestAdd (0.00s)
PASS mathutils.TestMultiply (0.00s)
PASS mathutils.Test_AddFlaky (0.00s)
PASS Package mathutils (450ms)
DONE 5 tests, 1 failure in 1.011s
::group::FAIL cmd.Test_Flaky2 (re-run 1) (0.00s)
main_test.go:21: Flaky test failed!
::endgroup::
FAIL Package cmd (184ms)
DONE 2 runs, 6 tests, 2 failures in 1.336s
PASS cmd.Test_Flaky2 (re-run 2) (0.00s)
PASS Package cmd (162ms)
=== Failed
=== FAIL: cmd Test_Flaky2 (0.00s)
main_test.go:21: Flaky test failed!
=== FAIL: cmd Test_Flaky2 (re-run 1) (0.00s)
main_test.go:21: Flaky test failed!
DONE 6 runs, 10 tests, 5 failures in 2.546s
```
I need a way to extract only the `=== Failed` section, if it exists, so that I can loop over the failed tests after this section. This approach would allow me to avoid iterating over all test cases for every package and would help identify flaky tests more quickly, especially when working on a large project with many test cases.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how gotestsum parses command output and how test_results.json is consumed, using the github-actions example and the `=== Failed` marker as the entry points. Reproduce output with and without that section; done means the failed-test loop can consume only that section without scanning every package.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100