huggingface / huggingface/datasets

Different objects are returned from calls that should be returning the same kind of object.

Open
#6,350 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

1. dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir, split='train[:1%]')
2. dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir)

The only difference I would expect these calls to have is the size of the dataset.

But, while 2. returns a dictionary with "train" key in it, 1. returns a dataset WITHOUT any initial "train" keyword.

Both calls are to be used within exactly the same context. They should return identically structured datasets of different size.

### Steps to reproduce the bug

See above.

### Expected behavior

Expect both calls to return the same structured Dataset structure but with different number of elements, i.e. call 1. should have 1% of the data of the call 2.0

### Environment info

Ubuntu 20.04
gcc 9.x.x.

It is really irrelevant.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.