Azure / Azure/azure-sdk-tools

[Eval framework] Enforce skill, MCP tool, and mock-handler eval coverage

Open
#16,345 0 comments 0 reactions 1 assignee Claimed by @helen229 View on GitHub
EngSys
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

Parent epic: #16344

## Goal

Make eval coverage enforceable: every supported skill and every public Azure SDK MCP tool must ship with meaningful starter coverage, and every mock-tier tool reference must have a realistic mock handler.

## Current baseline

The repository currently has 12 skill directories with `SKILL.md`; 8 have eval files and 4 do not:

- `azsdk-common-api-review`
- `azsdk-common-generate-sdk-locally`
- `sdk-ai-bot-eval-dataset`
- `sdk-ai-bot-run-evaluation`

## Scope

- Add a validator under `eng/common/scripts/eval/` that inventories skills, public MCP tools, eval references, and mock handlers.
- Require each supported skill to have at least one trigger/routing eval.
- Require each public MCP tool to be referenced by at least one tool-scenario stimulus.
- Detect stale eval tool names and missing mock handlers.
- Support documented exclusions for skills/tools evaluated by a different framework.
- Run schema/lint validation for discovered eval files.
- Emit actionable errors with the missing path/tool and scaffolding guidance.
- Add unit tests and wire the validator into PR validation.
- Roll out warn-only for existing quality defects, then make missing coverage blocking.

## Acceptance criteria

- [ ] New `SKILL.md` without required eval coverage fails validation.
- [ ] New public MCP tool without a tool eval fails validation.
- [ ] Mock-tier eval referencing a tool without a mock handler fails validation.
- [ ] Stale eval references are reported.
- [ ] Exclusions are explicit, reviewed, and documented.
- [ ] Current uncovered skills are covered or intentionally excluded.
- [ ] Validator has unit tests and runs in CI.

## Related

Native sub-issues:

- [x] #15852 ΓÇö MCP tool/mock-handler drift
- [ ] #15871 ΓÇö Per-skill eval quality
- [ ] #15874 ΓÇö Skill-eval authoring guidance
- [ ] #15875 ΓÇö Per-skill eval lint gate

- [ ] #16403 — Document current eval coverage and gaps

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.