Dataset Health & Drift Analysis Tool for EvalAI / CloudCV

Nobody has claimed this yet.

Hello EvalAI / CloudCV maintainers,

My name is Jobin Sabu. While exploring EvalAI workflows, I noticed that dataset quality checks (duplicates, corruption, imbalance, drift) are largely handled manually before training or evaluation.

I’m currently building a standalone tool tentatively called Dataset Drifter, focused on dataset health and drift analysis for computer vision datasets. The goal is to help users identify dataset issues early, before model training or challenge submissions.

Initial scope (MVP)

Detect duplicate and near-duplicate images

Identify corrupted or unreadable files

Analyze class distribution and imbalance

Flag outliers using embedding-based similarity

Generate a concise dataset health report (CLI + basic visual summary)

Design goals

Modular, extensible architecture

Works with common CV dataset formats (image folders + annotations)

Usable as a CLI or Python library

Potentially integrable with EvalAI / CloudCV workflows in the future

I’m completing a working MVP first and wanted to check:

whether this direction aligns with EvalAI / CloudCV goals, and

if there are any existing tools, preferred formats, or constraints I should align with early.

Once the MVP is ready, I’d be happy to share the repository and discuss possible integration or next steps.

Thank you for maintaining EvalAI and CloudCV — it’s a very valuable platform for the community.

Best regards,
Jobin Sabu

No repository files, tests, or entry points are identified. Start by reviewing EvalAI/CloudCV workflows and the proposed MVP scope, then determine whether an integration point and supported dataset formats can be defined. Done would require an agreed direction or shared working MVP, rather than a specified repository change.

Written by the indexing model from the issue text.