project-chip / project-chip/matter-test-scripts
Implement per-job single retry of the failed tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19
- Forks
- 7
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 3
Description
Background: Flaky tests are a known source of instability in our validation workflows, especially in the REPL and Darwin jobs. These transient failures (such as network errors or timeouts due to a slow runner) cause entire jobs to fail for reasons unrelated to the code changes, forcing developers to manually intervene. This wastes time and consumes unnecessary compute resources.
Proposal: Implement a selective, automated retry mechanism. Instead of failing the entire job on the first error, the workflow will capture the names of failed tests and re-run only those specific tests. The job will be marked as successful if the retry passes and will only fail if the same tests fail a second time. Flakiness data will be captured as a build artifact for long-term analysis.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Inspect the repository's validation workflow definitions, focusing on the REPL and Darwin jobs, and first determine how failed test names and job results are currently produced. Done means only failed tests receive one retry, the job fails after a second failure, and flakiness data is saved as a build artifact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100