Complete the eval gate: completion regression, rerun, and multiple-comparison policies
- Dominant language
- C#
- Stars
- 5.4k
- Forks
- 415
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 84
Description
## Current scope
PR #1027 completed the practical-significance work from this issue:
- [x] Exact one-sided sign test
- [x] 20% task-level net-win floor
- [x] Failed-slot judge retry and frozen successful judgments
- [x] Exact fail-closed result accounting
The remaining work is intentionally narrower:
- [ ] Add a trusted objective-completion regression hard gate when deterministic grader identity is available.
- [ ] Define the rerun policy so repeated runs cannot weaken the nominal significance guarantee.
- [ ] Define the multiple-comparison policy for suite-wide evaluation.
The completion gate remains deferred because the current infrastructure cannot identify a trusted deterministic completion grader. Scenario-level aggregation and any threshold changes must be back-tested before they become merge gates.
Related implementation: #1027.
_(Copilot, updating this issue on Abhitej's behalf.)_
Contributor guide
Research direction
Start by reviewing related PR #1027 and the current evaluation infrastructure. Determine how a trusted deterministic completion grader could be identified, then define rerun and suite-wide multiple-comparison policies; done requires these remaining policies and the completion regression gate to be specified and back-tested before becoming merge gates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100