microsoft / microsoft/hve-core

fix: plan-validator eval stimuli reference nonexistent fixtures and pass on prompt echo

Open
#2,444 0 comments 1 reaction 1 assignee Claimed by @vdstrizhkova View on GitHub
bug needs-triage
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

## Component

Custom Agents

## Bug Description

The `plan-validator-discrepancy-log` stimulus in `evals/agent-behavior/stimuli/plan-validator.yml` does not meaningfully test the plan-validator's discrepancy-detection behavior, for two compounding reasons:

1. **Dangling fixtures** — the prompt tells the agent to validate `.copilot-tracking/plans/example.md` against `.copilot-tracking/research/example.md`, but neither file exists in the repo (`.copilot-tracking/` is gitignored, and no `example.md` fixture is committed anywhere). The agent has no real plan or research to compare, so no validation reasoning is exercised.
2. **Echo-passable graders** — the prompt already contains the exact tokens the graders match ("Discrepancy Log", "DR-", "DD-", `.copilot-tracking/plans`). The `discrepancy-log-vocabulary` and `planning-log-path` graders can therefore pass merely by the agent echoing the prompt — even on a "files not found" reply that produces zero discrepancy findings (a false positive).

The same dangling-`example.md` pattern also appears in `plan-validator-coverage-matrix` (framed as "describe how you build…" to sidestep the missing file) and in `evals/agent-behavior/stimuli/rpi-validator.yml`.

## Expected Behavior

The stimulus should exercise real behavior: supply an inline (or fixture-backed) plan and research snippet that genuinely contradict each other, and graders should require the agent to surface a discrepancy that the prompt does **not** pre-name — so a pass evidences actual detection rather than vocabulary echo.

## Steps to Reproduce

1. Open `evals/agent-behavior/stimuli/plan-validator.yml`.
2. Note the `plan-validator-discrepancy-log` prompt references `.copilot-tracking/plans/example.md` and `.copilot-tracking/research/example.md`.
3. Search the repo for `example.md` — no such fixture exists.
4. Compare the grader regexes to the prompt text — the matched tokens ("Discrepancy Log", "DR-", "DD-", planning-log path) are already present in the prompt.
5. Observe that a "cannot find the files" response still satisfies both graders.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.