Data quality level
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 264
- Avg merge
- 1d 13m
- Merged PRs (30d)
- 23
Description
We could add a data quality level either to BaseDataset or to Evaluation. The idea would be something like:
Level 0 (All data): Load everything that is defined in the dataset.
Level 1 (Sane data): Load data where the basic information corresponds to the description in the documentation/paper. E.g. only load subjects where Target/NonTarget ratios are correct and the number of events etc. is correct.
Level 2 (Hand-vetted data): Probably a lot of effort, but in this setting we could omit subjects / sessions where a large number of artifacts is produced e.g. due to eye blinks or faulty electrodes.
Level -1 (undefined): New dataset PRs need to be vetted, until that is done we could assign some arbitrary level to these.
Additionally, after a dataset has been checked, e.g. via a sanity check script, it could get a badge / tag on the documentation data checked and loading correctly and other datasets get either data not checked, use at your own risk or there are issues with this dataset, see Github #X
I am open to other suggestions / ideas.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the BaseDataset and Evaluation entry points mentioned in the issue, along with how dataset documentation currently represents validation status. Determine whether the project wants quality levels, documentation badges, or both. Done would require an agreed design, an implementation plan, and corresponding documentation for checked, unchecked, and known-problematic datasets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100