microsoft / microsoft/hve-core
feat(evals): complete Vally integration - CI workflow, presets, coverage, and tooling
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 301
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 92
Description
Tracking issue for completing Vally evaluation framework integration after the initial infrastructure landed in #1590.
## Current state
- `.vally.yaml` at repo root with 3 suites (skill-quality, agent-behavior, script-validation)
- `evals/` directory with 3 `eval.yaml` files, 8 stimuli testing 4 skills
- `@microsoft/vally-cli` 0.4.0 in devDependencies
- `eval:lint` npm script wired into `lint:all`
- `evals/results/` gitignored
- No CI workflow, no `eval:run`/`eval:compare` scripts, no devcontainer install
## Phase 1: tooling foothold (parallel, ~1 PR each)
### npm scripts
Add suite-specific aliases following the existing `lint:*` naming pattern:
```json
"eval:run": "npx vally eval",
"eval:run:skills": "npx vally eval --suite skill-quality",
"eval:run:agents": "npx vally eval --suite agent-behavior",
"eval:run:scripts": "npx vally eval --suite script-validation",
"eval:compare": "npx vally compare"
```
Use `npx vally` (not bare `vally`) so the script works without global install. Consider `eval:run:fast` once an inner-loop equivalent is wired (tag filter `cost=free` works today as a stopgap).
### Devcontainer / copilot-setup-steps
- `copilot-setup-steps.yml` already runs `npm ci`, so `@microsoft/vally-cli` is available via devDependencies. The gap is really `on-create.sh` which never runs `npm ci`. Fix: add `npm ci` to `post-create.sh` (faster container rebuilds than `on-create.sh`).
- Keep vally project-local via `npx` rather than global install to avoid version drift from lockfile.
- Add a `Verify tool availability` step: `npx vally --version`.
## Phase 2: shared environments in `.vally.yaml`
Highest-leverage config change - eliminates repeated `../../.github/skills/...` relative paths and is a prerequisite for expanding coverage (otherwise each new eval doubles path-maintenance burden).
Schema supports `environments` map with `skills`, `files`, `commands`, `git`, `mcpServers`. Also supports `paths.evalFilenames` for custom eval file naming patterns. Move repeated skill path lists into environments and reference by name from each `eval.yaml`.
## Phase 3: `evaluate.yml` CI workflow
Model after the upstream Vally `evaluate.yml` pattern. Should come after npm scripts/env wiring so the workflow consumes the same surface contributors use locally.
### Design decisions
- Triple-trigger pattern (`pull_request` + `pull_request_target` + `issue_comment`) with `pr-status`/`fork-pr-status`/`gate`/`evaluate`/`comment-on-pr`/`report-status` job split
- Change detection paths: `evals/`, `.vally.yaml`, `.github/skills/`, `.github/agents/`, `collections/`, `plugins/`, `.github/workflows/evaluate.yml`
- Commit-status context: `evaluation-status` (survives workflow renames)
- Concurrency group keyed to PR number with `cancel-in-progress: true`
- Permission gate: require `admin`/`write`/`maintain` to trigger `/evaluate`
- CLI invocation: `npx vally eval --suite ci --runs 3 --workers 5 --threshold 0 --junit --judge-model --timeout `
- Reporting: JUnit via `dorny/test-reporter@v3` + concatenated `eval-results.md` comment
- Cache `~/.npm` and `node_modules` keyed on `package-lock.json`
- `timeout-minutes: 120`, `retention-days: 30`
- Model pinning at workflow level (eval YAMLs stay model-agnostic; workflow pins the canary via `--judge-model`)
- Fork-PR security: follow `pull_request_target` + base-branch-checkout pattern (don't checkout PR head with secrets in scope)
### Additional CLI flags to consider
The CLI now supports these flags that may be useful in our workflow:
- `--judge-model` - pin the grading model separately from the eval target model
- `--timeout` - per-eval timeout (prevents runaway evals from blocking CI)
- `--max-retries` - retry transient failures without failing the suite
- `--grader-plugin` - custom grader plugins (for future spike work)
- `--executor-plugin` - custom executor plugins
- `--skip-grade` / `--skip-validate` - useful for debugging eval execution without grading
- `--output` - explicit output directory control
## Phase 4: coverage expansion (split into per-family issues)
Each family gets its own issue - one stimulus PR should be reviewable in one sitting with its own threshold/weight tuning.
- [ ] Security skills (owasp-top-10, owasp-cicd, secops)
- [ ] Code review skills (code-review, idiomatic-audit)
- [ ] Language standards (python, powershell, bash, csharp, rust)
- [ ] Prompt/instruction authoring (prompt-builder, writing-style)
- [ ] Design thinking (dt-coach, dt-curriculum)
- [ ] Agent behavior: routing vs boundary vs multi-skill vs handoff (4 sub-issues)
## Deferred (upstream-blocked)
### Preset references
`--preset` doesn't exist in `@microsoft/vally-cli` 0.4.0. The eval command only supports `--eval-spec`, `--suite`, and `--tag`. Presets in the upstream `presets/` directory are reference configurations without a CLI flag to load them automatically.
Upstream tracking: [microsoft/evaluate#362](https://github.com/microsoft/evaluate/issues/362)
Workaround: spell out flags explicitly (`--suite ci --runs 3 --workers 5 --threshold 0 --junit`).
### Project-level scoring weights
Not supported by `.vally.yaml` schema. Top-level properties are `paths`, `environments`, `suites` only. Per-grader-type weights exist on the eval spec (`scoring.weights: { : }`) but the scorer doesn't apply them yet (plain average).
Upstream tracking:
- [microsoft/evaluate#204](https://github.com/microsoft/evaluate/issues/204) - apply configured scoring weights in eval aggregation
- [microsoft/evaluate#158](https://github.com/microsoft/evaluate/issues/158) - per-grader-instance weights (beyond type-level)
Workaround: set weights inside each `eval.yaml` `scoring.weights:` block (schema-valid, not yet applied by scorer).
### Grader plugin exploration
Scope as a time-boxed spike, not a delivery item: (1) list candidate custom graders (response-length, structured-output, skill-activation), (2) prototype one as local plugin via `--grader-plugin ./scripts/graders/...`, (3) measure signal lift on existing stimuli. Don't ship until step 3 shows value.
## Open decisions
- [ ] Token strategy: single `COPILOT_GITHUB_TOKEN` or multi-token rotation (`_2/_3/_4`) for rate-limit spreading?
- [ ] Cost budget: expected cost per PR / per nightly? Workflow kill switch (`if: vars.EVAL_ENABLED == 'true'`)?
- [ ] Baseline storage for `vally compare`: artifact on `main`? release branch? external store?
- [ ] Result trends: artifact retention only, `vally export` to structured store, or JUnit history via dorny?
- [ ] `eval:lint` strictness: add `--strict` flag for CI?
- [ ] `type: regression` validity: confirm schema supports it before authoring regression suite (documented type is `capability`)
- [ ] `--judge-model` selection: which model for grading? Same as eval target or separate?
- [ ] `--timeout` value: what's a reasonable per-eval timeout for CI?
Contributor guide
Assessment
This issue has not been assessed yet.