huggingface / huggingface/datasets

Support parallelized downloading and processing in load_dataset with Spark

Open
#5,798 17 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Feature request

When calling `load_dataset` for datasets that have multiple files, support using Spark to distribute the downloading and processing job to worker nodes when `cache_dir` is a cloud file system shared among nodes.

```python
load_dataset(..., use_spark=True)
```

### Motivation

Further speed up `dl_manager.download` and `_prepare_split` by distributing the workloads to worker nodes.

### Your contribution

I can submit a PR to support this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.