huggingface / huggingface/datasets
Support cloud storage in load_dataset
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
Would be nice to be able to do
```python
data_files=["s3://..."] # or gs:// or any cloud storage path
storage_options = {...}
load_dataset(..., data_files=data_files, storage_options=storage_options)
```
The idea would be to use `fsspec` as in `download_and_prepare` and `save_to_disk`.
This has been requested several times already. Some users want to use their data from private cloud storage to train models
related:
https://github.com/huggingface/datasets/issues/3490
https://github.com/huggingface/datasets/issues/5244
[forum](https://discuss.huggingface.co/t/how-to-use-s3-path-with-load-dataset-with-streaming-true/25739/2)
Contributor guide
Assessment
This issue has not been assessed yet.