huggingface / huggingface/datasets

Support cloud storage in load_dataset

Open
#5,281 31 comments 61 reactions 0 assignees View on GitHub
enhancement good second issue
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

Would be nice to be able to do

```python
data_files=["s3://..."] # or gs:// or any cloud storage path
storage_options = {...}
load_dataset(..., data_files=data_files, storage_options=storage_options)
```

The idea would be to use `fsspec` as in `download_and_prepare` and `save_to_disk`.

This has been requested several times already. Some users want to use their data from private cloud storage to train models

related:

https://github.com/huggingface/datasets/issues/3490
https://github.com/huggingface/datasets/issues/5244
[forum](https://discuss.huggingface.co/t/how-to-use-s3-path-with-load-dataset-with-streaming-true/25739/2)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.