dotnet / dotnet/skills

[skills-eval] Eval-instrument validity: underpowered/biased evals manufacture false P0 regressions

Open
#986 2 comments 0 reactions 2 assignees Claimed by @webreidi View on GitHub
Triaged
Dominant language
C#
Stars
5.4k
Forks
415
Avg merge
1d 4h
Merged PRs (30d)
84

Description

## Summary

Several `skills-eval` verdicts measure the **eval instrument**, not the skill. Cross-family re-runs on latest `main` (judge ≠ executor family) reversed multiple P0 "FIX-REGRESSION" verdicts that drove PR #972.

Evidence runs:
- android-tombstone: https://github.com/dotnet/skills/actions/runs/30862999210
- 6 sibling staged skills: https://github.com/dotnet/skills/actions/runs/30882049180

## Failure modes

1. **Underpowered (n=1 stimulus).** A single-stimulus eval yields one trial → the sign test is `p=0.5` **always** and can *never* pass, regardless of skill quality. Affects `check-bin-obj-clash`, `eval-performance`, `dotnet-aot-compat`.
2. **Direction-based sign-test amplification.** Many marginal "both responses correct, gap is small" losses aggregate into a "significant regression." Affects `migrate-dotnet9-to-dotnet10` (gpt: `p=0.004`, but every loss is a polish nitpick).
3. **Biased fixtures / graders / guardrails.** Fixture cross-contamination (models borrow data from sibling files), fabricated/unresolvable grader targets, and baseline-unfailable guardrails. Affects `android-tombstone-symbolication` (#893).
4. **Construct-invalid stimuli.** Stimuli that don't exercise the skill. `msbuild-antipatterns`: 3 of 4 stimuli are F# file-ordering tasks, not anti-pattern review.

## Terminology (to prevent future confusion)

- **stimulus** = one scenario in `stimuli:` (coverage/breadth).
- **runs** = replays of each stimulus (`runs:`, default 1).
- **trials** = `stimuli × runs` — the unit the sign test counts.
- Sign-test power comes from **distinct stimuli**, not repeated runs. Repeated runs of one stimulus are correlated pseudo-replications: they denoise but don't add real n.

## Impact

PR #972 staged 8 skills on these verdicts; re-analysis shows **0–1** are warranted on the original premise. The rest are eval repairs.

## Related

Sibling of #909 (judge-side transient failures corrupting verdicts). This issue covers the *structural* instrument defects (power, sign-test direction, fixtures, construct validity).

## Tracking

Per-skill fixes tracked in the plugin reports: #893, #896, #899, #901, #895. First remediation (android-tombstone) in progress.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.