huggingface / huggingface/datasets

Inconsistent loading of LFS-hosted files in epfml/FineWeb-HQ dataset

Open
#7,882 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

Some files in the `epfml/FineWeb-HQ` dataset fail to load via the Hugging Face `datasets` library.

- xet-hosted files load fine
- LFS-hosted files sometimes fail

Example:
- Fails: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-26/000_00003.parquet
- Works: https://huggingface.co/datasets/epfml/FineWeb-HQ/blob/main/data/CC-MAIN-2024-42/000_00027.parquet

Discussion: https://huggingface.co/datasets/epfml/FineWeb-HQ/discussions/2

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset(
"epfml/FineWeb-HQ",
data_files="data/CC-MAIN-2024-26/000_00003.parquet",
)
```

Error message:
```
HfHubHTTPError: 403 Forbidden: None.
Cannot access content at: https://cdn-lfs-us-1.hf.co/repos/...
Make sure your token has the correct permissions.
...
AccessDeniedAccess Denied
```

### Expected behavior

It should load the dataset for all files.

### Environment info

- python 3.10
- datasets 4.4.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.