microsoft / microsoft/hve-core

Add agent routing and tool interception to the eval harness

Open
#2,642 0 comments 1 reaction 1 assignee View on GitHub

@WilliamBerryiii is already working on this.

Since Aug 9, 2026.

enhancement priority-3 testing
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

## Context

The agent-behavior eval suite cannot test whether an agent's tool grant is enforced at runtime. The harness exercises skill behavior under agent-style prompts; it does not load the agent artifact under test, because the runner has no agent routing. A stimulus that appears to test tool authority is testing the base model's response to a prompt, not the agent's actual capability boundary.

PR #2602 replaced five contrived agent stimuli with canonical structural conformance checks drawn from the repository's own eval-authoring standard (`.github/skills/hve-core/vally-tests/references/agents.md`). Those checks verify documented artifact contracts, which the harness can do. They do not and cannot assert tool enforcement.

## Why this was deferred

A reviewer proposed instrumented fake tools with forbidden-call assertions. That requires the runner to invoke an agent with its declared tool grant and intercept tool calls. Neither capability exists. This is a harness feature, not a test fix.

## Proposed work

Add agent routing and tool interception to the eval harness.

## Acceptance criteria

- [ ] The runner can invoke a named agent artifact with its declared tool grant applied
- [ ] Tool calls made during a run are recorded and assertable
- [ ] A stimulus can assert that a forbidden tool was not called
- [ ] A stimulus can assert that a required tool was called
- [ ] The agent conformance check catalog is extended with the runtime checks this unlocks
- [ ] The five agents added by PR #2602 gain enforcement-grade stimuli alongside their structural ones

## Notes

Until this exists, no stimulus in this repository should claim to demonstrate tool enforcement. The stimuli added by PR #2602 carry explicit negate graders rejecting that claim.

## Evidence

- PR #2602 review threads covering the agent-behavior eval boundary
- `.github/skills/hve-core/vally-tests/references/agents.md` — the nine agent conformance checks, none covering runtime tool enforcement
- `evals/agent-behavior/eval.yaml` — generated spec; its own description records that it exercises skill behavior under agent-style prompts

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.