demml / demml/opsml

[eval] CI: compare eval run against stored baseline

Open
#388 0 comments 0 reactions 0 assignees View on GitHub
Backend enhancement
Dominant language
Rust
Stars
40
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Part of #380
Depends on: #383, #384

## What

Logic for CI workflows where a team runs evaluations on the current version and compares against a previously stored run.

- Python API: `compare(current_result, baseline)` where baseline can be specified as:
- Service version (e.g. `service_version="1.2.0"`)
- Space/name/version triple
- Specific eval run uid
- Server route to fetch a stored eval result by the above identifiers
- Comparison output: per-metric delta, pass/fail regression verdict
- CLI: `opsml eval compare --baseline-version 1.2.0` for use in CI scripts
- Should return a non-zero exit code on regression (for CI gate use)

Contributor guide

Open the contributing guide

Research direction

Start by reading #380 and the dependent issues #383 and #384, then trace the Python compare(current_result, baseline) API, the server route for stored results, and the opsml eval CLI entry point. Done means supporting all listed baseline identifiers, reporting per-metric deltas and regression verdicts, and returning a non-zero exit code for CI regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, ci-cd, cli, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.