microsoft / microsoft/hve-core
Add agent routing and tool interception to the eval harness
@WilliamBerryiii is already working on this.
Since Aug 9, 2026.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 301
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 92
Description
## Context
The agent-behavior eval suite cannot test whether an agent's tool grant is enforced at runtime. The harness exercises skill behavior under agent-style prompts; it does not load the agent artifact under test, because the runner has no agent routing. A stimulus that appears to test tool authority is testing the base model's response to a prompt, not the agent's actual capability boundary.
PR #2602 replaced five contrived agent stimuli with canonical structural conformance checks drawn from the repository's own eval-authoring standard (`.github/skills/hve-core/vally-tests/references/agents.md`). Those checks verify documented artifact contracts, which the harness can do. They do not and cannot assert tool enforcement.
## Why this was deferred
A reviewer proposed instrumented fake tools with forbidden-call assertions. That requires the runner to invoke an agent with its declared tool grant and intercept tool calls. Neither capability exists. This is a harness feature, not a test fix.
## Proposed work
Add agent routing and tool interception to the eval harness.
## Acceptance criteria
- [ ] The runner can invoke a named agent artifact with its declared tool grant applied
- [ ] Tool calls made during a run are recorded and assertable
- [ ] A stimulus can assert that a forbidden tool was not called
- [ ] A stimulus can assert that a required tool was called
- [ ] The agent conformance check catalog is extended with the runtime checks this unlocks
- [ ] The five agents added by PR #2602 gain enforcement-grade stimuli alongside their structural ones
## Notes
Until this exists, no stimulus in this repository should claim to demonstrate tool enforcement. The stimuli added by PR #2602 carry explicit negate graders rejecting that claim.
## Evidence
- PR #2602 review threads covering the agent-behavior eval boundary
- `.github/skills/hve-core/vally-tests/references/agents.md` — the nine agent conformance checks, none covering runtime tool enforcement
- `evals/agent-behavior/eval.yaml` — generated spec; its own description records that it exercises skill behavior under agent-style prompts
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.