hoangsonww / hoangsonww/AegisVision-Computer-Vision-System

Feature: Scenario replay lab for regression-testing perception behavior

Open
#14 0 comments 0 reactions 1 assignee Claimed by @hoangsonww View on GitHub
bug documentation enhancement good first issue help wanted question
Dominant language
Go
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary
Add a scenario replay lab that can turn previously captured events, claim-check frame references, and pipeline revisions into deterministic replay suites for validating perception behavior before rollout.

## Problem / Opportunity
AegisVision has canary, shadow inference, load tests, and drift/SLO monitoring, but there is no dedicated workflow for replaying named real-world scenarios against candidate pipeline/model revisions. Operators need a way to ask whether a proposed revision still handles known difficult cases such as occlusion, lighting changes, dense scenes, or zone-boundary edge cases before it reaches production traffic.

## Proposed Feature
Introduce a replay workflow that lets users curate scenarios from historical events, bind them to the exact stream/pipeline/model context that produced them, and execute those scenarios against a candidate pipeline revision or model version. Results should compare detections, tracks, rule firings, latency, and relevant SLO deltas against the baseline revision.

## Scope
- Add a scenario resource model with tenant, stream, time window, event IDs, frame URNs, baseline pipeline revision, and expected outcomes.
- Add replay job APIs and persistence for queued/running/completed runs.
- Reuse claim-check references without putting frame bytes on the bus.
- Produce per-scenario diffs for detections, track continuity, rule events, and latency.
- Add console surfaces for creating scenarios from events and viewing replay results.
- Add docs explaining when replay should be used versus canary or shadow inference.

## Acceptance Criteria
- [ ] A user can create a named scenario from a stream/time window or selected event IDs.
- [ ] A user can run a scenario against a candidate pipeline revision or model version without mutating production state.
- [ ] Replay output includes baseline-versus-candidate diffs for events, detections, track IDs, and p50/p95 latency.
- [ ] Replay jobs are tenant-scoped, audited, and exposed through the API gateway.
- [ ] The console can create, run, and inspect replay scenarios.
- [ ] Documentation describes replay data retention, claim-check usage, and limitations.

## Non-Goals
- Replacing canary promotion or shadow inference.
- Storing raw frame bytes in Postgres or on the event bus.
- Guaranteeing bit-for-bit deterministic GPU inference across all hardware.

## Dependencies / Risks
- Needs careful integration with claim-check lifecycle and media retention.
- Replay may require bounded resource scheduling so large jobs do not starve live inference.
- Scenario expectations must avoid becoming brittle when model classes evolve.

## Open Questions
- Should replay jobs run through dataplane-runner, a separate replay-runner, or a special inference-router mode?
- Should scenarios support manually annotated expected outcomes in the first version?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.