[skills-eval] Eval-instrument validity: underpowered/biased evals manufacture false P0 regressions
- Dominant language
- C#
- Stars
- 5.4k
- Forks
- 415
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 84
Description
## Summary
Several `skills-eval` verdicts measure the **eval instrument**, not the skill. Cross-family re-runs on latest `main` (judge ≠ executor family) reversed multiple P0 "FIX-REGRESSION" verdicts that drove PR #972.
Evidence runs:
- android-tombstone: https://github.com/dotnet/skills/actions/runs/30862999210
- 6 sibling staged skills: https://github.com/dotnet/skills/actions/runs/30882049180
## Failure modes
1. **Underpowered (n=1 stimulus).** A single-stimulus eval yields one trial → the sign test is `p=0.5` **always** and can *never* pass, regardless of skill quality. Affects `check-bin-obj-clash`, `eval-performance`, `dotnet-aot-compat`.
2. **Direction-based sign-test amplification.** Many marginal "both responses correct, gap is small" losses aggregate into a "significant regression." Affects `migrate-dotnet9-to-dotnet10` (gpt: `p=0.004`, but every loss is a polish nitpick).
3. **Biased fixtures / graders / guardrails.** Fixture cross-contamination (models borrow data from sibling files), fabricated/unresolvable grader targets, and baseline-unfailable guardrails. Affects `android-tombstone-symbolication` (#893).
4. **Construct-invalid stimuli.** Stimuli that don't exercise the skill. `msbuild-antipatterns`: 3 of 4 stimuli are F# file-ordering tasks, not anti-pattern review.
## Terminology (to prevent future confusion)
- **stimulus** = one scenario in `stimuli:` (coverage/breadth).
- **runs** = replays of each stimulus (`runs:`, default 1).
- **trials** = `stimuli × runs` — the unit the sign test counts.
- Sign-test power comes from **distinct stimuli**, not repeated runs. Repeated runs of one stimulus are correlated pseudo-replications: they denoise but don't add real n.
## Impact
PR #972 staged 8 skills on these verdicts; re-analysis shows **0–1** are warranted on the original premise. The rest are eval repairs.
## Related
Sibling of #909 (judge-side transient failures corrupting verdicts). This issue covers the *structural* instrument defects (power, sign-test direction, fixtures, construct validity).
## Tracking
Per-skill fixes tracked in the plugin reports: #893, #896, #899, #901, #895. First remediation (android-tombstone) in progress.
Contributor guide
Assessment
This issue has not been assessed yet.