hoangsonww / hoangsonww/AegisVision-Computer-Vision-System

Feature: Dataset quality gates for annotation and training readiness

Open
#20 0 comments 0 reactions 1 assignee Claimed by @hoangsonww View on GitHub
bug documentation enhancement good first issue help wanted question
Dominant language
Go
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary
Add dataset quality gates that evaluate whether a dataset version is ready for annotation review, training, canary planning, or compliance export.

## Problem / Opportunity
The platform includes datasets, annotations, training jobs, active learning, lineage, and model promotion, but it lacks a structured readiness check for dataset quality. Without gates, teams can train or evaluate against datasets with class imbalance, duplicate samples, stale labels, missing consent metadata, poor inter-annotator agreement, or weak coverage of important scenes.

## Proposed Feature
Introduce dataset quality profiles and gate checks that compute readiness scores for dataset versions. Gates should be configurable per tenant/project and integrated into training-orchestrator, active-learning, model-registry, and console workflows.

## Scope
- Define dataset quality dimensions: coverage, class balance, label completeness, label agreement, duplicate/near-duplicate rate, consent/retention metadata, and scenario coverage.
- Add APIs to run quality checks and persist gate results per dataset version.
- Integrate training job creation so policies can warn or refuse based on gate status.
- Add console quality reports with drilldowns to sample cohorts needing attention.
- Add docs for interpreting scores and configuring tenant/project thresholds.
- Add tests using synthetic datasets with known quality failures.

## Acceptance Criteria
- [ ] A dataset version can be evaluated against a named quality profile.
- [ ] Results include pass/warn/fail status, score breakdown, and affected sample references.
- [ ] Training job creation can require passing quality gates for selected projects.
- [ ] Active-learning queues can prioritize samples that improve failed quality dimensions.
- [ ] Console users can inspect quality reports and navigate to problematic sample cohorts.
- [ ] Documentation includes recommended default thresholds and caveats.

## Non-Goals
- Replacing human annotation review.
- Guaranteeing downstream model performance from dataset checks alone.
- Building a generic data catalog outside AegisVision's CV dataset domain.

## Dependencies / Risks
- Some dimensions may require expensive scans; results should be cached per dataset version.
- Agreement metrics depend on annotation workflow data being consistently captured.
- Refusal policies must be configurable to avoid blocking early experimentation.

## Open Questions
- Which quality dimensions should be required by default versus opt-in?
- Should gate failures route through policy-gate-service for override workflows?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.