Introduce a flag to retry test failures only if TEST_INFRASTRUCTURE_FAILURE_FILE is written
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
### Description of the problem / feature request:
Google's internal version of bazel, has a flag called --treat_test_infrastructure_failures_as_flaky, which retries tests up to 3 times (similar to Flaky=True). This only happens if the test output contains a file in the $TEST_INFRASTRUCTURE_FAILURE_FILE as defined in the [bazel docs](https://docs.bazel.build/versions/main/test-encyclopedia.html#initial-conditions).
### Feature requests: what underlying problem are you trying to solve with this feature?
The feature is trying to improve tests which may have flaky infrastructure, but are generally reliable otherwise. For example, we have some tests which relying on starting an emulator for testing. If this emulator fails to start, we'd like to treat that as an infrastructure failure and indicate the test should be retried.
This is an improvement over using Flaky=True on the test target, because there can be other test failures we would prefer not to be retried. `--treat_test_infrastructure_failures_as_flaky` would give tests the ability to have flaky retries only during certain [infrastructure] failures.
Contributor guide
Research direction
The issue names no source files or tests; begin by locating Bazel's test-result handling and existing retry or flaky-test flag entry points. Compare the requested TEST_INFRASTRUCTURE_FAILURE_FILE behavior with the linked Bazel test encyclopedia guidance; done means the new flag retries only infrastructure-marked failures while leaving other failures unretried.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100