huggingface / huggingface/datasets
Support parallelized downloading and processing in load_dataset with Spark
Open
enhancement
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Feature request
When calling `load_dataset` for datasets that have multiple files, support using Spark to distribute the downloading and processing job to worker nodes when `cache_dir` is a cloud file system shared among nodes.
```python
load_dataset(..., use_spark=True)
```
### Motivation
Further speed up `dl_manager.download` and `_prepare_split` by distributing the workloads to worker nodes.
### Your contribution
I can submit a PR to support this.
Contributor guide
Assessment
This issue has not been assessed yet.