hoangsonww / hoangsonww/AegisVision-Computer-Vision-System
Feature: Scenario replay lab for regression-testing perception behavior
- Dominant language
- Go
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Add a scenario replay lab that can turn previously captured events, claim-check frame references, and pipeline revisions into deterministic replay suites for validating perception behavior before rollout.
## Problem / Opportunity
AegisVision has canary, shadow inference, load tests, and drift/SLO monitoring, but there is no dedicated workflow for replaying named real-world scenarios against candidate pipeline/model revisions. Operators need a way to ask whether a proposed revision still handles known difficult cases such as occlusion, lighting changes, dense scenes, or zone-boundary edge cases before it reaches production traffic.
## Proposed Feature
Introduce a replay workflow that lets users curate scenarios from historical events, bind them to the exact stream/pipeline/model context that produced them, and execute those scenarios against a candidate pipeline revision or model version. Results should compare detections, tracks, rule firings, latency, and relevant SLO deltas against the baseline revision.
## Scope
- Add a scenario resource model with tenant, stream, time window, event IDs, frame URNs, baseline pipeline revision, and expected outcomes.
- Add replay job APIs and persistence for queued/running/completed runs.
- Reuse claim-check references without putting frame bytes on the bus.
- Produce per-scenario diffs for detections, track continuity, rule events, and latency.
- Add console surfaces for creating scenarios from events and viewing replay results.
- Add docs explaining when replay should be used versus canary or shadow inference.
## Acceptance Criteria
- [ ] A user can create a named scenario from a stream/time window or selected event IDs.
- [ ] A user can run a scenario against a candidate pipeline revision or model version without mutating production state.
- [ ] Replay output includes baseline-versus-candidate diffs for events, detections, track IDs, and p50/p95 latency.
- [ ] Replay jobs are tenant-scoped, audited, and exposed through the API gateway.
- [ ] The console can create, run, and inspect replay scenarios.
- [ ] Documentation describes replay data retention, claim-check usage, and limitations.
## Non-Goals
- Replacing canary promotion or shadow inference.
- Storing raw frame bytes in Postgres or on the event bus.
- Guaranteeing bit-for-bit deterministic GPU inference across all hardware.
## Dependencies / Risks
- Needs careful integration with claim-check lifecycle and media retention.
- Replay may require bounded resource scheduling so large jobs do not starve live inference.
- Scenario expectations must avoid becoming brittle when model classes evolve.
## Open Questions
- Should replay jobs run through dataplane-runner, a separate replay-runner, or a special inference-router mode?
- Should scenarios support manually annotated expected outcomes in the first version?
Contributor guide
Assessment
This issue has not been assessed yet.