microsoft / microsoft/hve-core
test(evals): add Experiment Designer lifecycle evidence and fix negated-grader denial defect
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 301
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 92
Description
## Summary
Two RAI findings against the Experiment Designer agent have no behavioral evidence:
- `RAI-P08-G01-A019-C01`
- `RAI-P08-G02-A079-C01`
Both are currently `NOT_EXECUTED`. This issue tracks adding merge-gating agent-behavior scenarios that exercise the Experiment Designer lifecycle, plus the grader correctness work that surfaced while building them.
## Scope
### 1. Experiment Designer lifecycle evidence
Add six agent-behavior scenarios covering the lifecycle the findings are about:
| Scenario | Covers |
|---|---|
| supported classification | problem-class routing on a supported input |
| ambiguous classification | refusal to over-commit on an unclear input |
| progressive hypotheses | hypothesis formation across phases |
| weak evidence rollback | phase rollback when evidence is insufficient |
| backlog consent | no backlog writes without explicit consent |
| failed tracking write | honest reporting when a tracking write fails |
Supported by synthetic fixtures under `evals/agent-behavior/fixtures/experiment-lifecycle/` with unique preservation markers, so context-loss regressions are detectable.
### 2. Negated-grader denial defect
While building the above, a defect was found across the agent-behavior suite.
Graders written as "fail if the agent claims it did X" also match the agent stating it did **not** do X. Because `negate: true` inverts the match, an agent answering honestly — *"I have not modified package.json"* — **fails** the scenario, while a vague answer passes. The gate rewarded silence and penalised accurate reporting.
Scope of the defect:
- 38 `no-source-edit` graders, 27 of them merge-gating
- ~17 further negated graders in other families, mostly via a bare `successfully` alternative, so *"the board was not successfully updated"* also counted as a false claim
## Acceptance criteria
- [ ] Six Experiment Designer scenarios exist and are merge-gating (no `tags.advisory`)
- [ ] Generated `eval.yaml` shows no drift from its stimulus partials
- [ ] Negated graders no longer match truthful denials, with no loss of genuine detection
- [ ] Eval lint lanes pass: vally, safety, schema
- [ ] Scenarios execute in trusted CI so both findings stop being `NOT_EXECUTED`
## Notes
Local execution of the scenarios is not possible without `COPILOT_GITHUB_TOKEN`. Until CI runs them, the deliverable is validated test *definitions* plus the grader fix — not behavioral evidence. The final acceptance criterion is what closes that gap.
Related: #2935 (EV-05 / EV-06 destination evidence) — some affected graders originate from that work.
Contributor guide
Assessment
This issue has not been assessed yet.