Azure / Azure/azure-sdk-tools

[Eval framework] Document current eval coverage and gaps

Open
#16,403 0 comments 0 reactions 1 assignee Claimed by @helen229 View on GitHub
EngSys
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

Parent workstream: #16345
Parent epic: #16344

## Goal

Publish and maintain a stakeholder-readable inventory of Azure SDK Agent eval coverage so teams can see what is tested today, identify gaps, and prioritize new scenarios.

## Scope

- Inventory current eval surfaces: MCP tool-selection, workflow scenarios, skill trigger/routing, skill capability, mock, and live tiers.
- Map covered skills, public MCP tools, workflow scenarios, repositories, and test tiers.
- Distinguish deterministic coverage checks from model-graded behavioral coverage.
- Document known gaps, exclusions, owners, and the issue tracking each gap.
- Include completed versus pending framework capabilities and links to representative pipeline results.
- Define how the inventory stays current as skills/tools/scenarios are added.
- Share the document with the Shanga team, Laurent, Justin, and other eval stakeholders for review.

## Deliverable

A checked-in document in `azure-sdk-tools` linked from the eval README and epic #16344, with tables for:

1. Eval type and purpose.
2. Skills and scenarios currently covered.
3. MCP tool and mock-handler coverage.
4. Cross-repository coverage.
5. Known gaps and follow-up issues.

## Acceptance criteria

- [ ] Current eval types and tiers are documented.
- [ ] Covered skills, tools, and workflow scenarios are listed.
- [ ] Missing/partial coverage is explicit and linked to issues.
- [ ] Specs and language-repo rollout status is visible.
- [ ] Stakeholder review feedback is incorporated.
- [ ] A maintenance owner/process is documented.

## Related

- #16345 — automated coverage enforcement
- #15871 — per-skill eval quality
- #16347 — specs-repo pilot
- #16348 — language-repo rollout

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.