[Eval framework] Enforce skill, MCP tool, and mock-handler eval coverage
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
Parent epic: #16344
## Goal
Make eval coverage enforceable: every supported skill and every public Azure SDK MCP tool must ship with meaningful starter coverage, and every mock-tier tool reference must have a realistic mock handler.
## Current baseline
The repository currently has 12 skill directories with `SKILL.md`; 8 have eval files and 4 do not:
- `azsdk-common-api-review`
- `azsdk-common-generate-sdk-locally`
- `sdk-ai-bot-eval-dataset`
- `sdk-ai-bot-run-evaluation`
## Scope
- Add a validator under `eng/common/scripts/eval/` that inventories skills, public MCP tools, eval references, and mock handlers.
- Require each supported skill to have at least one trigger/routing eval.
- Require each public MCP tool to be referenced by at least one tool-scenario stimulus.
- Detect stale eval tool names and missing mock handlers.
- Support documented exclusions for skills/tools evaluated by a different framework.
- Run schema/lint validation for discovered eval files.
- Emit actionable errors with the missing path/tool and scaffolding guidance.
- Add unit tests and wire the validator into PR validation.
- Roll out warn-only for existing quality defects, then make missing coverage blocking.
## Acceptance criteria
- [ ] New `SKILL.md` without required eval coverage fails validation.
- [ ] New public MCP tool without a tool eval fails validation.
- [ ] Mock-tier eval referencing a tool without a mock handler fails validation.
- [ ] Stale eval references are reported.
- [ ] Exclusions are explicit, reviewed, and documented.
- [ ] Current uncovered skills are covered or intentionally excluded.
- [ ] Validator has unit tests and runs in CI.
## Related
Native sub-issues:
- [x] #15852 ΓÇö MCP tool/mock-handler drift
- [ ] #15871 ΓÇö Per-skill eval quality
- [ ] #15874 ΓÇö Skill-eval authoring guidance
- [ ] #15875 ΓÇö Per-skill eval lint gate
- [ ] #16403 — Document current eval coverage and gaps
Contributor guide
Assessment
This issue has not been assessed yet.