huggingface / huggingface/datasets
Different objects are returned from calls that should be returning the same kind of object.
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
1. dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir, split='train[:1%]')
2. dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir)
The only difference I would expect these calls to have is the size of the dataset.
But, while 2. returns a dictionary with "train" key in it, 1. returns a dataset WITHOUT any initial "train" keyword.
Both calls are to be used within exactly the same context. They should return identically structured datasets of different size.
### Steps to reproduce the bug
See above.
### Expected behavior
Expect both calls to return the same structured Dataset structure but with different number of elements, i.e. call 1. should have 1% of the data of the call 2.0
### Environment info
Ubuntu 20.04
gcc 9.x.x.
It is really irrelevant.
Contributor guide
Assessment
This issue has not been assessed yet.