dotnet / dotnet/skills

Complete the eval gate: completion regression, rerun, and multiple-comparison policies

Open
#970 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
5.4k
Forks
415
Avg merge
1d 4h
Merged PRs (30d)
84

Description

## Current scope

PR #1027 completed the practical-significance work from this issue:

- [x] Exact one-sided sign test
- [x] 20% task-level net-win floor
- [x] Failed-slot judge retry and frozen successful judgments
- [x] Exact fail-closed result accounting

The remaining work is intentionally narrower:

- [ ] Add a trusted objective-completion regression hard gate when deterministic grader identity is available.
- [ ] Define the rerun policy so repeated runs cannot weaken the nominal significance guarantee.
- [ ] Define the multiple-comparison policy for suite-wide evaluation.

The completion gate remains deferred because the current infrastructure cannot identify a trusted deterministic completion grader. Scenario-level aggregation and any threshold changes must be back-tested before they become merge gates.

Related implementation: #1027.

_(Copilot, updating this issue on Abhitej's behalf.)_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing related PR #1027 and the current evaluation infrastructure. Determine how a trusted deterministic completion grader could be identified, then define rerun and suite-wide multiple-comparison policies; done requires these remaining policies and the completion regression gate to be specified and back-tested before becoming merge gates.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.