Kaszanas / Kaszanas/SC2_Datasets

Validator Compatibility Table, Final Validator Role

Open
#69 1 comment 0 reactions 1 assignee Claimed by @Kaszanas View on GitHub
enhancement feature request
Dominant language
Python
Stars
17
Forks
4
PR merge metrics
No merged PRs in 30d

Description

### Is your proposal related to a problem?

The dataset interfaces using single JSON file as the input, might have to implement different validation logic.
For example, current integrity validators just reading the JSON files and seeing if they are not corrupted might not work for a dataset where the files are now JSON objects within a larger JSON array.

They are no longer files but objects of a lazily loaded huge JSON file.

### Describe the solution you'd like

Define a proper way of integrating validation logic for all kinds of datasets that there are.
Currently validators act like filters, they can exclude specific samples from training. This logic should be described further and solidified.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.