JesseRWeigel / JesseRWeigel/no-flaky-test

Validate rule precision on an independently labeled real-test corpus

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Priority

P2 — focused reliability or product improvement

## Context

Portfolio audit follow-up (2026-09-08). The useful product claim is actionable detection, not a high finding count. Existing hand-labeled examples should not be described as independent human validation unless reviewer provenance supports that statement.

## Implementation scope

Assemble a small licensed/pinned corpus across supported runners with held-out examples. Record expected behavior, rationale and review provenance separately from rule output. Include real async setup/cleanup and legitimate nondeterminism. Publish per-rule precision/recall or explicit unknowns, and decide recommended versus opt-in defaults from the evidence.

## Acceptance criteria

- [ ] Corpus sources, revisions, licenses and labeling method are documented.
- [ ] A held-out split prevents tuning and reporting on only the same examples.
- [ ] False positives and false negatives are reported per rule with reproducible commands.
- [ ] Every recommended rule has actionable example diagnostics and a documented acceptance threshold.
- [ ] Claims distinguish automated labels, maintainer labels and independent human review.

Keep the change focused on this issue. Use synthetic fixtures for sensitive data and mocked providers for automated tests; do not require paid model calls to run the regression suite.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the six ESLint rules, their supported runners, and the automated regression tests. Define the licensed, pinned corpus, held-out split, labeling provenance, and reproducible reporting before tuning rules. Done means per-rule false positives and false negatives, actionable diagnostics, acceptance thresholds, and documented default recommendations without paid model calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
eslint, javascript
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.