microsoft / microsoft/hve-core

feat(evals): establish VEX promotion gate with eval loop and labeled CVE corpus

Open
#2,132 1 comment 1 reaction 0 assignees View on GitHub
enhancement evals experimental security
Dominant language
Python
Stars
1.5k
Forks
301
Avg merge
3d 3h
Merged PRs (30d)
92

Description

## Summary

The VEX capability ships at `experimental` maturity. Its core determination — asserting a CVE is `not_affected` — is produced by an LLM-assisted agent and is therefore non-deterministic. Before we promote VEX to `stable` and let downstream scanners suppress findings based on it, we need a measurable accuracy baseline backed by an evaluation loop and a labeled corpus of CVEs.

This issue tracks building that eval coverage and defining the numeric promotion gate.

Relates to #2038, #1220, #1221.

## Why

* `not_affected` suppresses real scanner findings; a false `not_affected` is a silent miss with security consequences.
* The capability has no eval coverage today (`evals/` contains no VEX/OpenVEX suites).
* The repo now has first-class eval support (`vally`), so this is the right place to encode the promotion gate as a repeatable, CI-runnable check.

## Scope

- [ ] Define a **labeled CVE corpus** with ground-truth status per CVE (`not_affected` / `affected` / `fixed` / `under_investigation`), including the justification code for `not_affected` cases.
- [ ] Add a VEX eval suite under `evals/agent-behavior/` exercising the `vex-generator` agent + `cve-analyzer` subagent against the corpus.
- [ ] Use executor `copilot-sdk` with `runs: 3+` to account for non-determinism; record per-run agreement.
- [ ] Capture the **false-positive rate on `not_affected`** as the headline metric (a wrong `not_affected` is the costly error).
- [ ] Wire the suite into eval CI per `docs/contributing/evals-ci.md` and `evals/README.md`.

## Promotion gate (proposed)

Promote VEX from `experimental` to `stable` only when the capability is validated across **3+ codebases** with a **≤5% false-positive rate on `not_affected` determinations**, measured over `runs: 3+`.

> Open question: the ≤5% figure is meaningless without the labeled baseline; sequence corpus definition **before** locking the numeric bar. Consider whether 5% is acceptable for a security-suppression control or should be tighter.

## Acceptance criteria

- [ ] Labeled CVE corpus exists and is documented (source, ground-truth method, size).
- [ ] VEX eval suite runs in CI and reports the `not_affected` false-positive rate.
- [ ] Promotion criteria are documented alongside the VEX collection-yml `maturity` entries.
- [ ] A clear, recorded decision on the numeric gate before any `experimental` → `stable` change.

## Notes

* Highest-priority follow-up from the PR #2038 review: it is the gate that lets us trust `not_affected` at scale.

Contributor guide

Open the contributing guide

Research direction

Read docs/contributing/evals-ci.md and evals/README.md first, then inspect evals/agent-behavior/ and the VEX collection-yml maturity entries. Define and document the labeled CVE corpus and the numeric promotion decision before adding the eval coverage. Done means the suite runs in CI, reports per-run agreement and the not_affected false-positive rate, and records the promotion criteria.

Written by the indexing model from the issue text.

Assessment

Domain
ci-cd, security, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.