huggingface / huggingface/datasets
Cannot use cached dataset without Internet connection (or when servers are down)
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
I want to be able to use cached dataset from HuggingFace even when I have no Internet connection (or when HuggingFace servers are down, or my company has network issues).
The problem why I can't use it:
`data_files` argument from `datasets.load_dataset()` function get it updates from the server before calculating hash for caching. As a result, when I run the same code with and without Internet I get different dataset configuration directory name.
### Steps to reproduce the bug
```
import datasets
c4_dataset = datasets.load_dataset(
path="allenai/c4",
data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
split="train",
cache_dir="/datesets/cache",
download_mode="reuse_cache_if_exists",
token=False,
)
```
1. Run this code with the Internet.
2. Run the same code without the Internet.
### Expected behavior
When running without the Internet connection, the loader should be able to get dataset from cache
### Environment info
- `datasets` version: 2.19.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.12.2
Contributor guide
Assessment
This issue has not been assessed yet.