microsoft / microsoft/hve-core
feat(evals): establish VEX promotion gate with eval loop and labeled CVE corpus
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 301
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 92
Description
## Summary
The VEX capability ships at `experimental` maturity. Its core determination — asserting a CVE is `not_affected` — is produced by an LLM-assisted agent and is therefore non-deterministic. Before we promote VEX to `stable` and let downstream scanners suppress findings based on it, we need a measurable accuracy baseline backed by an evaluation loop and a labeled corpus of CVEs.
This issue tracks building that eval coverage and defining the numeric promotion gate.
Relates to #2038, #1220, #1221.
## Why
* `not_affected` suppresses real scanner findings; a false `not_affected` is a silent miss with security consequences.
* The capability has no eval coverage today (`evals/` contains no VEX/OpenVEX suites).
* The repo now has first-class eval support (`vally`), so this is the right place to encode the promotion gate as a repeatable, CI-runnable check.
## Scope
- [ ] Define a **labeled CVE corpus** with ground-truth status per CVE (`not_affected` / `affected` / `fixed` / `under_investigation`), including the justification code for `not_affected` cases.
- [ ] Add a VEX eval suite under `evals/agent-behavior/` exercising the `vex-generator` agent + `cve-analyzer` subagent against the corpus.
- [ ] Use executor `copilot-sdk` with `runs: 3+` to account for non-determinism; record per-run agreement.
- [ ] Capture the **false-positive rate on `not_affected`** as the headline metric (a wrong `not_affected` is the costly error).
- [ ] Wire the suite into eval CI per `docs/contributing/evals-ci.md` and `evals/README.md`.
## Promotion gate (proposed)
Promote VEX from `experimental` to `stable` only when the capability is validated across **3+ codebases** with a **≤5% false-positive rate on `not_affected` determinations**, measured over `runs: 3+`.
> Open question: the ≤5% figure is meaningless without the labeled baseline; sequence corpus definition **before** locking the numeric bar. Consider whether 5% is acceptable for a security-suppression control or should be tighter.
## Acceptance criteria
- [ ] Labeled CVE corpus exists and is documented (source, ground-truth method, size).
- [ ] VEX eval suite runs in CI and reports the `not_affected` false-positive rate.
- [ ] Promotion criteria are documented alongside the VEX collection-yml `maturity` entries.
- [ ] A clear, recorded decision on the numeric gate before any `experimental` → `stable` change.
## Notes
* Highest-priority follow-up from the PR #2038 review: it is the gate that lets us trust `not_affected` at scale.
Contributor guide
Research direction
Read docs/contributing/evals-ci.md and evals/README.md first, then inspect evals/agent-behavior/ and the VEX collection-yml maturity entries. Define and document the labeled CVE corpus and the numeric promotion decision before adding the eval coverage. Done means the suite runs in CI, reports per-run agreement and the not_affected false-positive rate, and records the promotion criteria.
Written by the indexing model from the issue text.
Assessment
- Domain
- ci-cd, security, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100