demml / demml/opsml

[eval] Python + CLI: register eval scenarios

Open
#384 0 comments 0 reactions 0 assignees View on GitHub
Backend enhancement
Dominant language
Rust
Stars
40
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Part of #380
Depends on: #383

## What

Python-facing registration API and CLI command for eval scenarios.

- Extend `CardRegistry` (or add an eval-specific registry path) to support registering `EvalScenario` objects
- Support loading `EvalScenario` from a YAML/TOML file
- CLI: `opsml eval register --file eval.yaml` — reads scenario file, registers to server
- Registration should be idempotent on same scenario hash
- Works in both local and server mode

Contributor guide

Open the contributing guide

Research direction

Start with the existing CardRegistry and EvalScenario paths, then inspect the opsml eval register entry point and the scenario format used by eval.yaml. Trace dependency #383 before deciding where registration belongs. Done means YAML/TOML scenarios register through the CLI in local and server modes, with repeated registration of the same scenario hash producing no duplicate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.