gotestyourself / gotestyourself/gotestsum
Propose skip filter instead of run when retry failed tests
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 171
- PR merge metrics
- No merged PRs in 30d
Description
At the moment, when `--rerun-fails` is set, If some tests fail, the rerun is performed with a filter to only run the failed tests.
I propose to add another retry management, to `skip` passed tests instead of rerun only failed tests. In this way, it could also work if `failfast` option is set for example.
To use `skip` effectively, instead of skip only passed tests, we could add an option to define some group of tests that should be skipped if all of them passes.
My proposal is:
- add a `skip-at-prefix` option to test run, which enable skip mode and define the test prefix to skip
- add a `never-skip` option, if some tests should always be run because are useful for some test setup, for example in complex e2e testing
## skip-at-prefix
Imagine we have this test structure:
```
--- FAIL: TestSiblings
--- FAIL: TestSiblings/nested
--- FAIL: TestSiblings/nested/level_1
--- PASS TestSiblings/pass_1
```
If we set `--skip-at-prefix=TestSiblings/` option, the retry will have `-skip ^TestSiblings/pass_1`.
With the previous example, if go test run with `failfast` option, it will retry without skipping tests, since no test passed.
`skip-at-prefix` will skip by group, so skip will be at `Prefix+/subTestName`, for each test at that level inside that group. This could be interesting to avoid skipping test too specifically, but grouping by some logic by the user.
## never-skip
Imagine we have this test structure, and always want to exec `TestSetup` test also if passed because it prepare the test environment in some way.
```
--- PASS: TestSetup
--- FAIL: TestSiblings
--- FAIL: TestSiblings/nested
--- FAIL: TestSiblings/nested/level_1
--- PASS TestSiblings/pass_1
```
We could set `--never-skip=TestSetup`, and at each run it will be not added in the skipped tests.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the existing --rerun-fails retry behavior and how it interacts with Go test's -skip and failfast options. Use the proposed nested-subtest examples to define coverage for --skip-at-prefix and --never-skip, including passed setup tests; done means retries skip the intended groups while retaining failed tests and required setup tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100