[Eval framework] Document current eval coverage and gaps
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
Parent workstream: #16345
Parent epic: #16344
## Goal
Publish and maintain a stakeholder-readable inventory of Azure SDK Agent eval coverage so teams can see what is tested today, identify gaps, and prioritize new scenarios.
## Scope
- Inventory current eval surfaces: MCP tool-selection, workflow scenarios, skill trigger/routing, skill capability, mock, and live tiers.
- Map covered skills, public MCP tools, workflow scenarios, repositories, and test tiers.
- Distinguish deterministic coverage checks from model-graded behavioral coverage.
- Document known gaps, exclusions, owners, and the issue tracking each gap.
- Include completed versus pending framework capabilities and links to representative pipeline results.
- Define how the inventory stays current as skills/tools/scenarios are added.
- Share the document with the Shanga team, Laurent, Justin, and other eval stakeholders for review.
## Deliverable
A checked-in document in `azure-sdk-tools` linked from the eval README and epic #16344, with tables for:
1. Eval type and purpose.
2. Skills and scenarios currently covered.
3. MCP tool and mock-handler coverage.
4. Cross-repository coverage.
5. Known gaps and follow-up issues.
## Acceptance criteria
- [ ] Current eval types and tiers are documented.
- [ ] Covered skills, tools, and workflow scenarios are listed.
- [ ] Missing/partial coverage is explicit and linked to issues.
- [ ] Specs and language-repo rollout status is visible.
- [ ] Stakeholder review feedback is incorporated.
- [ ] A maintenance owner/process is documented.
## Related
- #16345 — automated coverage enforcement
- #15871 — per-skill eval quality
- #16347 — specs-repo pilot
- #16348 — language-repo rollout
Contributor guide
Assessment
This issue has not been assessed yet.