Measure flakiness of new tests
Nobody has claimed this yet.
- Dominant language
- Jinja
- Stars
- 541
- Forks
- 185
- Avg merge
- 22h 15m
- Merged PRs (30d)
- 4
Description
As discussed in https://github.com/nodejs/TSC/issues/1457, could we somehow have a way for CI to measure the flakiness of new tests before they land? Something like:
- For every PR, identify tests that are added by the PR (probably tests that run in the PR’s branch that didn’t run for
main). - Run
measure-flakinesson them. - Fail CI unless the new tests pass the flakiness cutoff, on all platforms.
This obviously won’t help for existing flaky tests, but I would think that it should prevent most new flaky tests from landing on main; and it would highly motivate contributors to improve their tests, because their PRs would be blocked from landing until they did so.
It also wouldn’t help if a test becomes flaky because of changes to the API that it tests after the test has landed. But still, I think this is better than the status quo.
Related: https://github.com/nodejs/build/pull/3056 cc @nodejs/tsc
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with the existing measure-flakiness command and the CI configuration for nodejs/build; inspect how PR-only tests are identified and how all-platform jobs report results. Done means new tests are selected, flakiness is measured on every platform, and CI blocks a PR when the cutoff is exceeded.
Written by the indexing model from the issue text.
Assessment
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100