huggingface / huggingface/datasets

Using load_dataset with data_files and split arguments yields an error

Open
#7,394 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

It seems the list of valid splits recorded by the package becomes incorrectly overwritten when using the `data_files` argument.

If I run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="all_examples", data_files="tasks/expert.jsonl")
```
then I get the error
```
ValueError: Unknown split "all_examples". Should be one of ['train'].
```

However, if I run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="train", name="Expert")
```
then I get
```
ValueError: Unknown split "train". Should be one of ['all_examples'].
```

### Steps to reproduce the bug

Run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="all_examples", data_files="tasks/expert.jsonl")
```

### Expected behavior

No error.

### Environment info

Python = 3.12
datasets = 3.2.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.