Add an azd ai eval extension for Foundry evaluations
- Dominant language
- Go
- Stars
- 569
- Forks
- 364
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 136
Description
### What
`azd` has no way to define or run Foundry evaluations. Everything an evaluation
needs a dataset, an evaluator, the eval that grades one with the other is
created by hand against the data plane, and nothing about it lives in the
project alongside the agent it grades.
### Proposal
An `azd ai eval` extension that puts evaluation config in the repo and
reconciles it like any other azd service:
```
azd ai eval init scaffold config for an agent, offline
azd ai eval generate generate a dataset and a rubric evaluator
azd ai eval create|list|show|delete
azd ai eval run start|list|show|cancel|delete
azd ai eval run output list|show|export
```
- `evals/azure.eval.yaml` declares datasets, evaluators and evals; `azure.yaml`
references it through an `azure.ai.eval` service entry.
- The service target reconciles datasets → evaluators → evals in dependency
order, publishing no redundant versions when nothing changed.
- Every command works standalone against `--project-endpoint`, so the extension
is usable outside an azd project.
Implemented by #9500.
Contributor guide
Assessment
This issue has not been assessed yet.