microsoft / microsoft/hve-core

feat(evals): partition planner conformance into skill-backed and native-agent execution

Open
#2,570 0 comments 1 reaction 0 assignees View on GitHub
needs-triage
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

## Issue Description

PR #2552 migrates the existing planner behavioral-conformance suites from beval to Vally so the repository can retire the beval execution path. During local validation, we found that the migrated multi-turn stimuli use a first turn such as:

```yaml
turns:
- Launch .github/agents/accessibility/accessibility-planner.agent.md
-
```

Which do not activate the named agent. Vally’s built-in copilot-sdk executor treats both entries as ordinary user prompts, so the default Copilot agent handles the scenario.

The 80 planner scenarios are not duplicates of behavior-conformance/skill-behavior.eval.yaml. Behavior conformance primarily uses regex graders to test skill knowledge, routing, and scope boundaries. Planner conformance uses semantic rubrics for domain decisions and orchestration behavior.

The repository is moving authoritative domain knowledge into skills, but the six .agent.md files remain shipped runtime artifacts. They still own tools, handoffs, startup sequencing, session state, phase transitions, confirmation gates, cross-planner routing, and skill loading.

Coverage should therefore be partitioned by behavior ownership rather than moved entirely to skills or retained under simulated agent-launch semantics.

PR #2552 should remain focused on retiring beval. This issue tracks the follow-up correctness work.

## Proposed Scope

### Classify Existing Scenarios

Inventory all 80 planner scenarios and classify each as:

- **Skill-owned:** Standards knowledge, framework mappings, domain heuristics, and reference material.
- **Agent-owned:** Startup sequencing, state, phase gates, handoffs, tools, routing, and coaching flow.
- **Hybrid:** Agent orchestration applying one or more skills.
- **Duplicate:** Routing or scope coverage already present in behavior conformance.
- **Retired:** No longer representative, with a documented rationale.

### Establish Skill-Backed Semantic Conformance

- Load authoritative dependencies through `environment.skills`.
- Replace synthetic `Launch ...agent.md` turns with direct scenario prompts.
- Preserve semantic rubrics that provide coverage beyond existing regex checks.
- Add missing semantic coverage for planner skills where appropriate.
- Consolidate only genuine duplicates with `skill-behavior.eval.yaml`.

### Establish Genuine Native-Agent Conformance

- Select a harness that genuinely activates `.agent.md`, such as:
- a Vally executor plugin using Copilot SDK custom-agent selection; or
- a VS Code-hosted test harness.
- Retain focused native-agent coverage for:
- startup disclaimers and framework attribution;
- session state and recovery;
- phase progression and confirmation gates;
- tools and handoffs;
- cross-planner routing;
- DT Coach conversational behavior and method transitions;
- correct on-demand skill loading.
- Document which VS Code host semantics the harness reproduces.
- Do not describe SDK emulation as full VS Code-host execution.

### Align Suites and CI

- Keep fast regex routing checks in behavior conformance.
- Keep semantic skill checks and native-agent checks distinct.
- Rename suites and documentation to state exactly what artifact is loaded.
- Determine advisory or authoritative status using the existing graduation policy.
- Update workflow and package entry points only where required.

## Acceptance Criteria

- [ ] All 80 planner scenarios are classified by behavior ownership or retired with rationale.
- [ ] No stimulus uses `Launch .agent.md` to claim custom-agent activation.
- [ ] Skill-owned scenarios load dependencies through `environment.skills`.
- [ ] Semantic coverage absent from behavior conformance is preserved.
- [ ] Genuine duplicate routing and knowledge checks are consolidated.
- [ ] Native-agent scenarios demonstrably activate the selected `.agent.md`.
- [ ] Raw evidence identifies the selected agent and loaded skills.
- [ ] Harness fidelity and VS Code-specific limitations are documented.
- [ ] Documentation distinguishes regex routing, skill-backed semantic evaluation, SDK agent emulation, and VS Code-host execution.
- [ ] Targeted Vally lint and schema validation passes.
- [ ] Resulting suites run successfully in their owning local or CI lanes.

## Non-Goals

- Expanding PR #2552 beyond retirement of beval.
- Reproducing unsupported VS Code UI behavior such as rendered handoff buttons.
- Reworking unrelated Vally infrastructure delivered by #1599.
- Adopting the separate agent-skill-bench proposal tracked by #1504.
- Retaining every scenario in native-agent execution solely because it originated in an agent suite.

## Additional Context

Current coverage:

- Planner conformance: 80 semantic scenarios.
- 30 DT Coach scenarios.
- 10 scenarios for each of the other five planners.
- Skill behavior conformance: primarily knowledge, tool-trigger, and bleed-detection scenarios using regex graders.

Official Vally documentation:

- [Eval specification](https://microsoft.github.io/vally/reference/eval-spec/)
- [Skill loading](https://microsoft.github.io/vally/reference/cli/eval/)
- [`SKILL.md` discovery](https://microsoft.github.io/vally/reference/skill-md/)
- [Custom executors](https://microsoft.github.io/vally/guides/writing-custom-executors/)

Related work:

- #2552
- #1599
- #1504

Contributor guide

Open the contributing guide

Research direction

Start with the 80 scenarios in the planner conformance suite and behavior-conformance/skill-behavior.eval.yaml, then read the Vally eval specification and skill-loading documentation. Classify each scenario as skill-owned, agent-owned, hybrid, duplicate, or retired; establish distinct skill-backed and genuinely native-agent suites, document harness limitations, and run targeted Vally lint, schema validation, and the owning local or CI lanes.

Written by the indexing model from the issue text.

Assessment

Tech stack
vscode, yaml
Domain
ci-cd, documentation, testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.