huggingface / huggingface/datasets
[Multi Configs] Use directories to differentiate between subsets/configurations
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
Currently to define several subsets/configurations of your dataset, you need to use a dataset script.
However it would be nice to have a no-code way to to this.
For example we could specify different configurations of a dataset (for example, if a dataset contains different languages) with one directory per configuration.
These structures are not supported right now, but would be nice to have:
```
my_dataset_repository/
├── README.md
├── en/
│ ├── train.csv
│ └── test.csv
└── fr/
├── train.csv
└── test.csv
```
Or with one directory per split:
```
my_dataset_repository/
├── README.md
├── en/
│ ├── train/
│ │ ├── shard_0.csv
│ │ └── shard_1.csv
│ └── test/
│ ├── shard_0.csv
│ └── shard_1.csv
└── fr/
├── train/
│ ├── shard_0.csv
│ └── shard_1.csv
└── test/
├── shard_0.csv
└── shard_1.csv
```
cc @stevhliu @albertvillanova
This can be specified in the README as YAML with
```
configs:
- config_name: en
data_dir: en
- config_name: fr
data_dir: fr
```
Contributor guide
Assessment
This issue has not been assessed yet.