Fix or quarantine the flaky tests that make CI cost double
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
A flaky test is paid for twice: once when the job fails and once on the retry, and it also teaches contributors to re-run rather than read a red check. With the runner budget now a hard constraint, flakiness is a budget problem, not only an annoyance.
Establish which tests are actually flaky from CI history, quarantine them out of the PR path, and fix them behind that. The migrated JIRAs #16740, #16207 and #15248 have partial lists from the Azure Pipelines era.
Part of #19524.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing CI history to establish which tests are flaky, then compare the partial lists in migrated JIRAs #16740, #16207, and #15248. Done means the identified tests are quarantined out of the PR path and fixes are made behind that quarantine, with CI cost and reliability improved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100