hoangsonww / hoangsonww/AegisVision-Computer-Vision-System
Feature: Dataset quality gates for annotation and training readiness
- Dominant language
- Go
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Add dataset quality gates that evaluate whether a dataset version is ready for annotation review, training, canary planning, or compliance export.
## Problem / Opportunity
The platform includes datasets, annotations, training jobs, active learning, lineage, and model promotion, but it lacks a structured readiness check for dataset quality. Without gates, teams can train or evaluate against datasets with class imbalance, duplicate samples, stale labels, missing consent metadata, poor inter-annotator agreement, or weak coverage of important scenes.
## Proposed Feature
Introduce dataset quality profiles and gate checks that compute readiness scores for dataset versions. Gates should be configurable per tenant/project and integrated into training-orchestrator, active-learning, model-registry, and console workflows.
## Scope
- Define dataset quality dimensions: coverage, class balance, label completeness, label agreement, duplicate/near-duplicate rate, consent/retention metadata, and scenario coverage.
- Add APIs to run quality checks and persist gate results per dataset version.
- Integrate training job creation so policies can warn or refuse based on gate status.
- Add console quality reports with drilldowns to sample cohorts needing attention.
- Add docs for interpreting scores and configuring tenant/project thresholds.
- Add tests using synthetic datasets with known quality failures.
## Acceptance Criteria
- [ ] A dataset version can be evaluated against a named quality profile.
- [ ] Results include pass/warn/fail status, score breakdown, and affected sample references.
- [ ] Training job creation can require passing quality gates for selected projects.
- [ ] Active-learning queues can prioritize samples that improve failed quality dimensions.
- [ ] Console users can inspect quality reports and navigate to problematic sample cohorts.
- [ ] Documentation includes recommended default thresholds and caveats.
## Non-Goals
- Replacing human annotation review.
- Guaranteeing downstream model performance from dataset checks alone.
- Building a generic data catalog outside AegisVision's CV dataset domain.
## Dependencies / Risks
- Some dimensions may require expensive scans; results should be cached per dataset version.
- Agreement metrics depend on annotation workflow data being consistently captured.
- Refusal policies must be configurable to avoid blocking early experimentation.
## Open Questions
- Which quality dimensions should be required by default versus opt-in?
- Should gate failures route through policy-gate-service for override workflows?
Contributor guide
Assessment
This issue has not been assessed yet.