microsoft / microsoft/GitHub-Copilot-for-Azure

CI: Wire vally eval invocation to a curated medium-size suite via .vally.yaml

Open
#1,920 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Context

Follow-up to [#1912](https://github.com/microsoft/GitHub-Copilot-for-Azure/pull/1912), which migrated Waza eval references to Vally. That PR deliberately wired `.github/workflows/eval.yml` to a single hardcoded `--eval-spec` path to keep scope small and get CI green.

## Goal

Replace the single `--eval-spec` flag with `--suite ` driven by `.vally.yaml` suite config, so CI runs a curated set of eval specs (not just one).

## Target

- **Inner loop (local dev)**: `vally-cli eval --eval-spec ` — fast iteration on a single spec.
- **Medium loop (CI)**: `vally-cli eval --suite pr` — curated set targeting 5–10 minute wall-clock runtime.
- **Outer loop (nightly / manual)**: `vally-cli eval --suite full` or individual dispatches — full coverage.

## Prereqs

- Tag hygiene pass (completed in #1912): `cost: free` now reflects reality, so suite filters behave correctly.
- Define `pr` and `full` suites in `.vally.yaml` with the right tag filters.

## Acceptance

- `.vally.yaml` defines `pr` and `full` suites with explicit tag filters
- `.github/workflows/eval.yml` Run evaluations step uses `--suite pr`
- CI wall-clock time for the `pr` suite is 5–10 minutes
- Results artifact still uploads at the standard path

## Related

- PR #1912 — initial Waza migration groundwork
- Thread: https://github.com/microsoft/GitHub-Copilot-for-Azure/pull/1912#discussion_r3096070310 (jongio's original comment)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing .github/workflows/eval.yml with the migration context in PR #1912, then inspect the existing or expected .vally.yaml configuration. Define the pr and full suites with explicit tag filters and update the Run evaluations step to use --suite pr. Confirm the results artifact remains at its standard path and that the pr suite runs in 5–10 minutes.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, yaml
Domain
ci-cd, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.