huggingface / huggingface/datasets

Add filename in error message when ReadError or similar occur

Open
#7,280 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

Please update error messages to include relevant information for debugging when loading datasets with `load_dataset()` that may have a few corrupted files.

Whenever downloading a full dataset, some files might be corrupted (either at the source or from downloading corruption).

However the errors often only let me know it was a tar file if `tarfile.ReadError` appears on the traceback, and I imagine similarly for other file types.

This makes it really hard to debug which file is corrupted, and when dealing with very large datasets, it shouldn't be necessary to force download everything again.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.