gotestyourself / gotestyourself/gotestsum

Detect Flaky tests in an optimized way

Open
#456 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Go
Stars
2.7k
Forks
171
PR merge metrics
No merged PRs in 30d

Description

Right now, I am trying to detect flaky tests, but I think looping over the test_results.json file will be inefficient since I would need to go through all packages. However, I realized that the output of the gotestsum command lists the different packages being tested, along with whether each test succeeded or failed. At the end of the output, there is a `=== Failed` section.

For example, the output in the case of the github-actions format might look like this:

```
::group::PASS cmd.TestStable (0.00s)
main_test.go:9: This is a stable test

::endgroup::
::group::FAIL cmd.Test_Flaky2 (0.00s)
main_test.go:21: Flaky test failed!

::endgroup::
FAIL Package cmd (297ms)

PASS mathutils.TestAdd (0.00s)
PASS mathutils.TestMultiply (0.00s)
PASS mathutils.Test_AddFlaky (0.00s)
PASS Package mathutils (450ms)

DONE 5 tests, 1 failure in 1.011s

::group::FAIL cmd.Test_Flaky2 (re-run 1) (0.00s)
main_test.go:21: Flaky test failed!

::endgroup::
FAIL Package cmd (184ms)

DONE 2 runs, 6 tests, 2 failures in 1.336s

PASS cmd.Test_Flaky2 (re-run 2) (0.00s)
PASS Package cmd (162ms)

=== Failed
=== FAIL: cmd Test_Flaky2 (0.00s)
main_test.go:21: Flaky test failed!

=== FAIL: cmd Test_Flaky2 (re-run 1) (0.00s)
main_test.go:21: Flaky test failed!

DONE 6 runs, 10 tests, 5 failures in 2.546s
```

I need a way to extract only the `=== Failed` section, if it exists, so that I can loop over the failed tests after this section. This approach would allow me to avoid iterating over all test cases for every package and would help identify flaky tests more quickly, especially when working on a large project with many test cases.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how gotestsum parses command output and how test_results.json is consumed, using the github-actions example and the `=== Failed` marker as the entry points. Reproduce output with and without that section; done means the failed-test loop can consume only that section without scanning every package.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.