nodejs / nodejs/node-core-utils
ncu-ci walk: exclude PR-modified tests from the flaky count
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 313
- Forks
- 137
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 24
Description
Current Behavior
Failures currently contribute to the flaky count even when the PR modified the failing test. Such failures are likely caused by the PR and should not count as independent flaky occurrences.
For example, CI build 75799 failed test-ffi-fast-buffer, but the PR modified test/ffi/test-ffi-fast-buffer.js. Excluding this occurrence would leave only one qualifying PR, so the test would not appear in the reliability report:
https://github.com/nodejs/reliability/blob/main/reports/2026-08-15.md
Expected Behavior
Exclude an occurrence from the flaky count when its test file was modified by the tested PR revision.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ncu-ci walk logic that calculates flaky counts and reliability reports, then trace how each occurrence is associated with the tested PR revision. Use the test-ffi-fast-buffer.js example and CI build 75799 as the reference case; done means occurrences whose test file was modified by the PR no longer contribute to the flaky count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100