demml / demml/opsml

[Epic] Agentic Evaluation Framework

Open
#380 0 comments 0 reactions 0 assignees View on GitHub
Backend enhancement UI
Dominant language
Rust
Stars
40
Forks
3
PR merge metrics
No merged PRs in 30d

Description

## Overview

End-to-end evaluation framework for agentic services. Covers the full lifecycle: define eval scenarios, register them to Scouter, run evaluations (manually or in CI), and inspect results in the UI.

## Scope

- `EvalScenario` definition format (YAML/Python)
- `EvalRegistryRecord` — versioned registry entry backed by Scouter
- Python API + CLI for registration and retrieval
- UI: eval scenario table + metric dashboard
- CI integration: compare current eval run against a stored baseline

## Sub-issues

- [ ] EvalScenario schema + EvalRegistryRecord data model
- [ ] Scouter write path for eval scenarios and results
- [ ] Python: register_eval_scenario() API
- [ ] CLI: register eval scenarios from file
- [ ] Python + CLI: list/pull registered eval scenarios
- [ ] UI: eval scenario table (paginated, editable)
- [ ] UI: evaluation metric dashboard
- [ ] CI: compare eval run against stored baseline

## Dependency order

`schema` → `write path` → `Python/CLI APIs` → `UI` → `CI integration`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.