microsoft / microsoft/hve-core

feat(evals): complete Vally integration - CI workflow, presets, coverage, and tooling

Open
#1,599 0 comments 0 reactions 1 assignee Claimed by @jongio View on GitHub
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

Tracking issue for completing Vally evaluation framework integration after the initial infrastructure landed in #1590.

## Current state

- `.vally.yaml` at repo root with 3 suites (skill-quality, agent-behavior, script-validation)
- `evals/` directory with 3 `eval.yaml` files, 8 stimuli testing 4 skills
- `@microsoft/vally-cli` 0.4.0 in devDependencies
- `eval:lint` npm script wired into `lint:all`
- `evals/results/` gitignored
- No CI workflow, no `eval:run`/`eval:compare` scripts, no devcontainer install

## Phase 1: tooling foothold (parallel, ~1 PR each)

### npm scripts

Add suite-specific aliases following the existing `lint:*` naming pattern:

```json
"eval:run": "npx vally eval",
"eval:run:skills": "npx vally eval --suite skill-quality",
"eval:run:agents": "npx vally eval --suite agent-behavior",
"eval:run:scripts": "npx vally eval --suite script-validation",
"eval:compare": "npx vally compare"
```

Use `npx vally` (not bare `vally`) so the script works without global install. Consider `eval:run:fast` once an inner-loop equivalent is wired (tag filter `cost=free` works today as a stopgap).

### Devcontainer / copilot-setup-steps

- `copilot-setup-steps.yml` already runs `npm ci`, so `@microsoft/vally-cli` is available via devDependencies. The gap is really `on-create.sh` which never runs `npm ci`. Fix: add `npm ci` to `post-create.sh` (faster container rebuilds than `on-create.sh`).
- Keep vally project-local via `npx` rather than global install to avoid version drift from lockfile.
- Add a `Verify tool availability` step: `npx vally --version`.

## Phase 2: shared environments in `.vally.yaml`

Highest-leverage config change - eliminates repeated `../../.github/skills/...` relative paths and is a prerequisite for expanding coverage (otherwise each new eval doubles path-maintenance burden).

Schema supports `environments` map with `skills`, `files`, `commands`, `git`, `mcpServers`. Also supports `paths.evalFilenames` for custom eval file naming patterns. Move repeated skill path lists into environments and reference by name from each `eval.yaml`.

## Phase 3: `evaluate.yml` CI workflow

Model after the upstream Vally `evaluate.yml` pattern. Should come after npm scripts/env wiring so the workflow consumes the same surface contributors use locally.

### Design decisions

- Triple-trigger pattern (`pull_request` + `pull_request_target` + `issue_comment`) with `pr-status`/`fork-pr-status`/`gate`/`evaluate`/`comment-on-pr`/`report-status` job split
- Change detection paths: `evals/`, `.vally.yaml`, `.github/skills/`, `.github/agents/`, `collections/`, `plugins/`, `.github/workflows/evaluate.yml`
- Commit-status context: `evaluation-status` (survives workflow renames)
- Concurrency group keyed to PR number with `cancel-in-progress: true`
- Permission gate: require `admin`/`write`/`maintain` to trigger `/evaluate`
- CLI invocation: `npx vally eval --suite ci --runs 3 --workers 5 --threshold 0 --junit --judge-model --timeout `
- Reporting: JUnit via `dorny/test-reporter@v3` + concatenated `eval-results.md` comment
- Cache `~/.npm` and `node_modules` keyed on `package-lock.json`
- `timeout-minutes: 120`, `retention-days: 30`
- Model pinning at workflow level (eval YAMLs stay model-agnostic; workflow pins the canary via `--judge-model`)
- Fork-PR security: follow `pull_request_target` + base-branch-checkout pattern (don't checkout PR head with secrets in scope)

### Additional CLI flags to consider

The CLI now supports these flags that may be useful in our workflow:

- `--judge-model` - pin the grading model separately from the eval target model
- `--timeout` - per-eval timeout (prevents runaway evals from blocking CI)
- `--max-retries` - retry transient failures without failing the suite
- `--grader-plugin` - custom grader plugins (for future spike work)
- `--executor-plugin` - custom executor plugins
- `--skip-grade` / `--skip-validate` - useful for debugging eval execution without grading
- `--output` - explicit output directory control

## Phase 4: coverage expansion (split into per-family issues)

Each family gets its own issue - one stimulus PR should be reviewable in one sitting with its own threshold/weight tuning.

- [ ] Security skills (owasp-top-10, owasp-cicd, secops)
- [ ] Code review skills (code-review, idiomatic-audit)
- [ ] Language standards (python, powershell, bash, csharp, rust)
- [ ] Prompt/instruction authoring (prompt-builder, writing-style)
- [ ] Design thinking (dt-coach, dt-curriculum)
- [ ] Agent behavior: routing vs boundary vs multi-skill vs handoff (4 sub-issues)

## Deferred (upstream-blocked)

### Preset references

`--preset` doesn't exist in `@microsoft/vally-cli` 0.4.0. The eval command only supports `--eval-spec`, `--suite`, and `--tag`. Presets in the upstream `presets/` directory are reference configurations without a CLI flag to load them automatically.

Upstream tracking: [microsoft/evaluate#362](https://github.com/microsoft/evaluate/issues/362)

Workaround: spell out flags explicitly (`--suite ci --runs 3 --workers 5 --threshold 0 --junit`).

### Project-level scoring weights

Not supported by `.vally.yaml` schema. Top-level properties are `paths`, `environments`, `suites` only. Per-grader-type weights exist on the eval spec (`scoring.weights: { : }`) but the scorer doesn't apply them yet (plain average).

Upstream tracking:
- [microsoft/evaluate#204](https://github.com/microsoft/evaluate/issues/204) - apply configured scoring weights in eval aggregation
- [microsoft/evaluate#158](https://github.com/microsoft/evaluate/issues/158) - per-grader-instance weights (beyond type-level)

Workaround: set weights inside each `eval.yaml` `scoring.weights:` block (schema-valid, not yet applied by scorer).

### Grader plugin exploration

Scope as a time-boxed spike, not a delivery item: (1) list candidate custom graders (response-length, structured-output, skill-activation), (2) prototype one as local plugin via `--grader-plugin ./scripts/graders/...`, (3) measure signal lift on existing stimuli. Don't ship until step 3 shows value.

## Open decisions

- [ ] Token strategy: single `COPILOT_GITHUB_TOKEN` or multi-token rotation (`_2/_3/_4`) for rate-limit spreading?
- [ ] Cost budget: expected cost per PR / per nightly? Workflow kill switch (`if: vars.EVAL_ENABLED == 'true'`)?
- [ ] Baseline storage for `vally compare`: artifact on `main`? release branch? external store?
- [ ] Result trends: artifact retention only, `vally export` to structured store, or JUnit history via dorny?
- [ ] `eval:lint` strictness: add `--strict` flag for CI?
- [ ] `type: regression` validity: confirm schema supports it before authoring regression suite (documented type is `capability`)
- [ ] `--judge-model` selection: which model for grading? Same as eval target or separate?
- [ ] `--timeout` value: what's a reasonable per-eval timeout for CI?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.