Azure / Azure/azure-dev

[evals] Add evals for copilot and azd in interactive mode

Open
#9,187 0 comments 0 reactions 1 assignee Claimed by @richardpark-msft View on GitHub
Dominant language
Go
Stars
569
Forks
364
Avg merge
2d 19h
Merged PRs (30d)
136

Description

(morphing from original description)

Copilot CLI, in the past, has had trouble driving azd in interactive mode. We want to add some evals to start recording our success rate and also the quality of the choices it makes (for instance, environment names, etc..)

Some possible trickiness:
- vally uses copilot SDK, which isn't quite the same as the CLI (but it uses the CLI underneath). It's probably good enough, but something to keep in mind in case.
- need to look into what other coding agents are supported by vally, as we'd want to cover Claude Code, etc...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.