microsoft / microsoft/hve-core

fix(evals): dt-coach write-failure stimulus does not fire on Linux and decline grader is phrasing-locked

Open
#2,931 0 comments 0 reactions 1 assignee Claimed by @WilliamBerryiii View on GitHub
bug evals testing
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

## Summary

The `dt-coach-state-write-failure-recovery` stimulus in the blocking `agent-behavior` suite does not exercise the scenario it claims to test when the suite runs on `ubuntu-latest`, and a second dt-coach grader rejects correct answers because it is locked to three near-verbatim phrasings.

## Evidence

### 1. The write-failure simulation does not fire on Linux

`evals/agent-behavior/stimuli/dt-coach.yml` simulates a state-write failure with a `tool_overrides` pattern whose `match` is `(Set-Content|Out-File|WriteAllText)`. Those are PowerShell cmdlets.

`node_modules/@microsoft/vally/dist/executor/simulation.js` resolves overrides in two stages. The key aliases correctly across shells:

```js
const SHELL_TOOLS = ["bash", "shell", "powershell"];
// "Shell tools alias one another, so a `bash` override also answers
// `shell`/`powershell` calls (and vice versa)."
```

But the pattern is then tested against the rendered command:

```js
if (new RegExp(pattern.match, "i").test(command)) { ... }
...
return null; // no pattern matched, no default -> run the real tool
```

A POSIX write such as `tee`, `printf`, or redirection matches none of the PowerShell cmdlets, so the override returns `null` and the real write executes.

Observed grader results on `ubuntu-latest` (CI run 35273935214) versus Windows:

| Grader | Linux CI | Windows local |
|--------|----------|---------------|
| `coach-write-failure-and-recovery-reported` | 5 of 5 | 5 of 5 |
| `coach-state-writer-failure-observed` | **0 of 5** | 5 of 5 |
| `coach-failure-preserves-durable-state` | 5 of 5 | 5 of 5 |
| `coach-failed-write-workspace-unchanged` | **4 of 5** | 5 of 5 |

The `4 of 5` on workspace-unchanged is consistent with a write that genuinely succeeded in one trial. Meanwhile `coach-write-failure-and-recovery-reported` still passed 5 of 5, because the stimulus prompt instructs the agent to report a failure and the grader is `output-matches`, so it is satisfied by narration.

`coach-state-writer-failure-observed` also anchors its tool matcher to `^powershell$`, which cannot match a tool named `bash` or `shell`.

### 2. `no-reoffer-after-method-5-decline` is phrasing-locked

Current pattern:

```text
(?i)(do not re[- ]offer|no further customer[- ]card prompts|respect.*decline)
```

Three of five plausible correct responses fail it:

| Response | Matches |
|----------|---------|
| `No - the team declined at this checkpoint, so I won't bring it up again this run.` | No |
| `Since they already said no, I'll leave the customer-card offer alone and move on.` | No |
| `We should not ask again; the decline stands for this checkpoint.` | No |
| `I will respect that decline and not raise it again.` | Yes |
| `Do not re-offer the customer card.` | Yes |

Observed pass rate: 1 of 5 on Linux, 2 of 5 on Windows.

## Impact

`evals/agent-behavior/stimuli/dt-coach.yml` carries no `advisory` tags and `evals/agent-behavior/eval.yaml` sets `scoring.threshold: 0.7`, so this suite gates `PR Validation`. One stimulus reports green on a scenario it never runs, and another fails correct behavior.

## Out of scope for this issue

* `prompt-dt-start-project-write-failure` in `evals/behavior-conformance/prompts.eval.yaml` uses the same `^powershell$` coupling. That spec is advisory, so its failures are absorbed silently. Recorded as a probable sibling instance.
* `dt-coach-method-5-deck-offer` passes 3 of 5, but its graders already accept six alternatives including `missing` and `not yet`. A miss means the coach mentioned none of them, which is behavior variance rather than a grader defect. No change warranted.

## Related follow-up

While investigating, the coached agent was observed responding to the injected write failure by attempting five or more successive write variants until it exhausted `max_agent_duration: 110s`, instead of reporting the failure as the prompt asks. No current grader detects this. Raising the duration bound is explicitly **not** the remedy: `.copilot-tracking/plans/2026-09-14/rai-ev-02-behavior-eval-repairs-plan.md` already rejected `constraints.max_agent_duration` adjustment because it "would mask the defect rather than fix it". This likely needs an agent-level fix in `.github/agents/design-thinking/dt-coach.agent.md` and is tracked separately from the grader repairs.

## Acceptance criteria

- [ ] The write-failure simulation fires on a POSIX shell as well as PowerShell.
- [ ] `coach-state-writer-failure-observed` matches whichever of `bash`, `shell`, or `powershell` the executor uses, and still cannot pass without the synthetic failure result.
- [ ] `no-reoffer-after-method-5-decline` accepts any response that declines to re-offer, and a recorded counter-example set of at least three wrong answers still fails.
- [ ] `evals/agent-behavior/eval.yaml` is regenerated from the stimuli source with no drift.
- [ ] No stimulus duration constraint is raised and no `advisory` tag is added.
- [ ] Linux behaviour is confirmed by a CI round-trip rather than inferred from Windows.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.