[Eval framework] Roll out the shared eval pipeline to Azure SDK language repositories
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
Parent epic: #16344
Depends on #16347 (successful specs-repo pilot) and #16346 (stable result contract).
## Goal
Roll out the shared Azure SDK Agent evaluation framework to every supported Azure SDK language repository using the existing `eng/common` and shared-skill sync mechanisms.
## Target repositories
- `azure-sdk-for-net`
- `azure-sdk-for-java`
- `azure-sdk-for-js`
- `azure-sdk-for-python`
- `azure-sdk-for-go`
- Other subscribed SDK repositories as agreed
## Scope
- Select one language repository as the pilot after specs.
- Verify synced eval templates/scripts and `.gitignore` behavior.
- Add thin repo-specific pipeline entrypoints and register ADO definitions.
- Configure pools, repository resources, variable groups, service connections, and auth.
- Add repo-specific pre/post hooks only where required.
- Add a minimal smoke eval per repo before language-specific scenarios.
- Start scheduled/report-only, monitor reliability/cost, then add selective PR path triggers.
- Define threshold and blocking policy per tier.
- Roll out through sync PRs rather than independent framework edits in each repository.
## Acceptance criteria
- [ ] One pilot language repo produces standard artifacts and summaries reliably.
- [ ] All agreed language repos have registered scheduled eval pipelines.
- [ ] Every repo emits repository/commit/run metadata in the common schema.
- [ ] Repo-specific setup is isolated behind hooks or thin entrypoints.
- [ ] Rollout, ownership, and incident response are documented.
- [ ] A decision is recorded for PR-gating policy in each tier.
Contributor guide
Assessment
This issue has not been assessed yet.