JesseRWeigel / JesseRWeigel/no-flaky-test
Validate rule precision on an independently labeled real-test corpus
- Dominant language
- JavaScript
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Priority
P2 — focused reliability or product improvement
## Context
Portfolio audit follow-up (2026-09-08). The useful product claim is actionable detection, not a high finding count. Existing hand-labeled examples should not be described as independent human validation unless reviewer provenance supports that statement.
## Implementation scope
Assemble a small licensed/pinned corpus across supported runners with held-out examples. Record expected behavior, rationale and review provenance separately from rule output. Include real async setup/cleanup and legitimate nondeterminism. Publish per-rule precision/recall or explicit unknowns, and decide recommended versus opt-in defaults from the evidence.
## Acceptance criteria
- [ ] Corpus sources, revisions, licenses and labeling method are documented.
- [ ] A held-out split prevents tuning and reporting on only the same examples.
- [ ] False positives and false negatives are reported per rule with reproducible commands.
- [ ] Every recommended rule has actionable example diagnostics and a documented acceptance threshold.
- [ ] Claims distinguish automated labels, maintainer labels and independent human review.
Keep the change focused on this issue. Use synthetic fixtures for sensitive data and mocked providers for automated tests; do not require paid model calls to run the regression suite.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the six ESLint rules, their supported runners, and the automated regression tests. Define the licensed, pinned corpus, held-out split, labeling provenance, and reproducible reporting before tuning rules. Done means per-rule false positives and false negatives, actionable diagnostics, acceptance thresholds, and documented default recommendations without paid model calls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- eslint, javascript
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100