[evals] Add evals for copilot and azd in interactive mode
Open
- Dominant language
- Go
- Stars
- 569
- Forks
- 364
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 136
Description
(morphing from original description)
Copilot CLI, in the past, has had trouble driving azd in interactive mode. We want to add some evals to start recording our success rate and also the quality of the choices it makes (for instance, environment names, etc..)
Some possible trickiness:
- vally uses copilot SDK, which isn't quite the same as the CLI (but it uses the CLI underneath). It's probably good enough, but something to keep in mind in case.
- need to look into what other coding agents are supported by vally, as we'd want to cover Claude Code, etc...
Contributor guide
Assessment
This issue has not been assessed yet.