huggingface / huggingface/datasets
EmptyDatasetError when loading dataset downloaded with HuggingFace cli
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I am using a cluster that does not have access to the internet when given a job. I tried downloading the dataset using the huggingface-cli command and then loading it with load_dataset but I get an error:
```raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None```
The dataset I'm using is "lmsys/chatbot_arena_conversations". The folder structure is
- README.md
- data
- train-00000-of-00001-cced8514c7ed782a.parquet
### Steps to reproduce the bug
1. Download dataset using HuggingFace CLI: ```huggingface-cli download lmsys/chatbot_arena_conversations --local-dir ./lmsys/chatbot_arena_conversations```
2. In Python
```
from datasets import load_dataset
load_dataset("lmsys/chatbot_arena_conversations")
```
### Expected behavior
Should return a Dataset Dict in the form of
```
DatasetDict({
train: Dataset({
features: [...],
num_rows: 33,000
})
})
```
### Environment info
Python 3.11.5
Datasets 2.18.0
Transformers 4.38.2
Pytorch 2.2.0
Pyarrow 15.0.1
Rocky Linux release 8.9 (Green Obsidian)
Contributor guide
Assessment
This issue has not been assessed yet.