gotestyourself / gotestyourself/gotestsum

Propose skip filter instead of run when retry failed tests

Open
#536 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.7k
Forks
171
PR merge metrics
No merged PRs in 30d

Description

At the moment, when `--rerun-fails` is set, If some tests fail, the rerun is performed with a filter to only run the failed tests.

I propose to add another retry management, to `skip` passed tests instead of rerun only failed tests. In this way, it could also work if `failfast` option is set for example.

To use `skip` effectively, instead of skip only passed tests, we could add an option to define some group of tests that should be skipped if all of them passes.

My proposal is:

- add a `skip-at-prefix` option to test run, which enable skip mode and define the test prefix to skip
- add a `never-skip` option, if some tests should always be run because are useful for some test setup, for example in complex e2e testing

## skip-at-prefix

Imagine we have this test structure:

```
--- FAIL: TestSiblings
--- FAIL: TestSiblings/nested
--- FAIL: TestSiblings/nested/level_1
--- PASS TestSiblings/pass_1
```

If we set `--skip-at-prefix=TestSiblings/` option, the retry will have `-skip ^TestSiblings/pass_1`.
With the previous example, if go test run with `failfast` option, it will retry without skipping tests, since no test passed.

`skip-at-prefix` will skip by group, so skip will be at `Prefix+/subTestName`, for each test at that level inside that group. This could be interesting to avoid skipping test too specifically, but grouping by some logic by the user.

## never-skip

Imagine we have this test structure, and always want to exec `TestSetup` test also if passed because it prepare the test environment in some way.

```
--- PASS: TestSetup

--- FAIL: TestSiblings
--- FAIL: TestSiblings/nested
--- FAIL: TestSiblings/nested/level_1
--- PASS TestSiblings/pass_1
```

We could set `--never-skip=TestSetup`, and at each run it will be not added in the skipped tests.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the existing --rerun-fails retry behavior and how it interacts with Go test's -skip and failfast options. Use the proposed nested-subtest examples to define coverage for --skip-at-prefix and --never-skip, including passed setup tests; done means retries skip the intended groups while retaining failed tests and required setup tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.