[eval] CI: compare eval run against stored baseline
- Dominant language
- Rust
- Stars
- 40
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
Part of #380
Depends on: #383, #384
## What
Logic for CI workflows where a team runs evaluations on the current version and compares against a previously stored run.
- Python API: `compare(current_result, baseline)` where baseline can be specified as:
- Service version (e.g. `service_version="1.2.0"`)
- Space/name/version triple
- Specific eval run uid
- Server route to fetch a stored eval result by the above identifiers
- Comparison output: per-metric delta, pass/fail regression verdict
- CLI: `opsml eval compare --baseline-version 1.2.0` for use in CI scripts
- Should return a non-zero exit code on regression (for CI gate use)
Contributor guide
Research direction
Start by reading #380 and the dependent issues #383 and #384, then trace the Python compare(current_result, baseline) API, the server route for stored results, and the opsml eval CLI entry point. Done means supporting all listed baseline identifiers, reporting per-metric deltas and regression verdicts, and returning a non-zero exit code for CI regressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- api, ci-cd, cli, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100