[Epic] Agentic Evaluation Framework
- Dominant language
- Rust
- Stars
- 40
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
## Overview
End-to-end evaluation framework for agentic services. Covers the full lifecycle: define eval scenarios, register them to Scouter, run evaluations (manually or in CI), and inspect results in the UI.
## Scope
- `EvalScenario` definition format (YAML/Python)
- `EvalRegistryRecord` — versioned registry entry backed by Scouter
- Python API + CLI for registration and retrieval
- UI: eval scenario table + metric dashboard
- CI integration: compare current eval run against a stored baseline
## Sub-issues
- [ ] EvalScenario schema + EvalRegistryRecord data model
- [ ] Scouter write path for eval scenarios and results
- [ ] Python: register_eval_scenario() API
- [ ] CLI: register eval scenarios from file
- [ ] Python + CLI: list/pull registered eval scenarios
- [ ] UI: eval scenario table (paginated, editable)
- [ ] UI: evaluation metric dashboard
- [ ] CI: compare eval run against stored baseline
## Dependency order
`schema` → `write path` → `Python/CLI APIs` → `UI` → `CI integration`
Contributor guide
Assessment
This issue has not been assessed yet.