huggingface / huggingface/datasets
Calling datasets.load_dataset("text" ...) results in a wrong split.
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
When creating a text dataset, the training split should have the bulk of the examples by default. Currently, testing does.
### Steps to reproduce the bug
I have a folder with 18K text files in it. Each text file essentially consists in a document or article scraped from online. Calling the following codeL
```
folder_path = "/home/cyril/Downloads/llama_dataset"
data = datasets.load_dataset("text", data_dir=folder_path)
data.save_to_disk("/home/cyril/Downloads/data.hf")
data = datasets.load_from_disk("/home/cyril/Downloads/data.hf")
print(data)
```
Results in the following split:
```
DatasetDict({
train: Dataset({
features: ['text'],
num_rows: 2114
})
test: Dataset({
features: ['text'],
num_rows: 200882
})
validation: Dataset({
features: ['text'],
num_rows: 152
})
})
```
It seems to me like the train/test/validation splits are in the wrong order since test split >>>> train_split
### Expected behavior
Train split should have the bulk of the training examples.
### Environment info
datasets 2.11.0, python 3.10.6
Contributor guide
Assessment
This issue has not been assessed yet.