microsoft / microsoft/GitHub-Copilot-for-Azure
CI: Wire vally eval invocation to a curated medium-size suite via .vally.yaml
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 250
- Forks
- 204
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 67
Description
## Context
Follow-up to [#1912](https://github.com/microsoft/GitHub-Copilot-for-Azure/pull/1912), which migrated Waza eval references to Vally. That PR deliberately wired `.github/workflows/eval.yml` to a single hardcoded `--eval-spec` path to keep scope small and get CI green.
## Goal
Replace the single `--eval-spec` flag with `--suite ` driven by `.vally.yaml` suite config, so CI runs a curated set of eval specs (not just one).
## Target
- **Inner loop (local dev)**: `vally-cli eval --eval-spec ` — fast iteration on a single spec.
- **Medium loop (CI)**: `vally-cli eval --suite pr` — curated set targeting 5–10 minute wall-clock runtime.
- **Outer loop (nightly / manual)**: `vally-cli eval --suite full` or individual dispatches — full coverage.
## Prereqs
- Tag hygiene pass (completed in #1912): `cost: free` now reflects reality, so suite filters behave correctly.
- Define `pr` and `full` suites in `.vally.yaml` with the right tag filters.
## Acceptance
- `.vally.yaml` defines `pr` and `full` suites with explicit tag filters
- `.github/workflows/eval.yml` Run evaluations step uses `--suite pr`
- CI wall-clock time for the `pr` suite is 5–10 minutes
- Results artifact still uploads at the standard path
## Related
- PR #1912 — initial Waza migration groundwork
- Thread: https://github.com/microsoft/GitHub-Copilot-for-Azure/pull/1912#discussion_r3096070310 (jongio's original comment)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing .github/workflows/eval.yml with the migration context in PR #1912, then inspect the existing or expected .vally.yaml configuration. Define the pr and full suites with explicit tag filters and update the Run evaluations step to use --suite pr. Confirm the results artifact remains at its standard path and that the pr suite runs in 5–10 minutes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, yaml
- Domain
- ci-cd, testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100