Azure / Azure/azure-dev

Add an azd ai eval extension for Foundry evaluations

Open
#9,549 0 comments 0 reactions 0 assignees View on GitHub
area/evals enhancement
Dominant language
Go
Stars
569
Forks
364
Avg merge
2d 19h
Merged PRs (30d)
136

Description

### What

`azd` has no way to define or run Foundry evaluations. Everything an evaluation
needs a dataset, an evaluator, the eval that grades one with the other is
created by hand against the data plane, and nothing about it lives in the
project alongside the agent it grades.

### Proposal

An `azd ai eval` extension that puts evaluation config in the repo and
reconciles it like any other azd service:

```
azd ai eval init scaffold config for an agent, offline
azd ai eval generate generate a dataset and a rubric evaluator
azd ai eval create|list|show|delete
azd ai eval run start|list|show|cancel|delete
azd ai eval run output list|show|export
```

- `evals/azure.eval.yaml` declares datasets, evaluators and evals; `azure.yaml`
references it through an `azure.ai.eval` service entry.
- The service target reconciles datasets → evaluators → evals in dependency
order, publishing no redundant versions when nothing changed.
- Every command works standalone against `--project-endpoint`, so the extension
is usable outside an azd project.

Implemented by #9500.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.