Azure / Azure/azure-sdk-tools

[Eval framework] Roll out the shared eval pipeline to Azure SDK language repositories

Open
#16,348 1 comment 0 reactions 1 assignee Claimed by @helen229 View on GitHub
EngSys
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

Parent epic: #16344

Depends on #16347 (successful specs-repo pilot) and #16346 (stable result contract).

## Goal

Roll out the shared Azure SDK Agent evaluation framework to every supported Azure SDK language repository using the existing `eng/common` and shared-skill sync mechanisms.

## Target repositories

- `azure-sdk-for-net`
- `azure-sdk-for-java`
- `azure-sdk-for-js`
- `azure-sdk-for-python`
- `azure-sdk-for-go`
- Other subscribed SDK repositories as agreed

## Scope

- Select one language repository as the pilot after specs.
- Verify synced eval templates/scripts and `.gitignore` behavior.
- Add thin repo-specific pipeline entrypoints and register ADO definitions.
- Configure pools, repository resources, variable groups, service connections, and auth.
- Add repo-specific pre/post hooks only where required.
- Add a minimal smoke eval per repo before language-specific scenarios.
- Start scheduled/report-only, monitor reliability/cost, then add selective PR path triggers.
- Define threshold and blocking policy per tier.
- Roll out through sync PRs rather than independent framework edits in each repository.

## Acceptance criteria

- [ ] One pilot language repo produces standard artifacts and summaries reliably.
- [ ] All agreed language repos have registered scheduled eval pipelines.
- [ ] Every repo emits repository/commit/run metadata in the common schema.
- [ ] Repo-specific setup is isolated behind hooks or thin entrypoints.
- [ ] Rollout, ownership, and incident response are documented.
- [ ] A decision is recorded for PR-gating policy in each tier.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.