When using a local cluster, shuffle with disk
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
In feedback for https://github.com/dask/dask/pull/8223, I've noticed users trying to do large shuffles on single-machine clusters: https://github.com/dask/dask/pull/8223#issuecomment-961610095, https://github.com/dask/dask/issues/8294#issuecomment-961301624.
When creating any default Client, the default shuffle mode automatically gets set to `"tasks"`:
https://github.com/dask/distributed/blob/69814b4aa7459476dcefa133341b566a5ed4e24a/distributed/client.py#L727-L730
However, on a single machine, a disk-based shuffle is likely to be a lot more efficient, plus much lower load on the scheduler.
I think it would be better to keep using the disk-based shuffle if the Client is connected to a LocalCluster (not sure how to tell this). Most users don't know about the different shuffle modes, and shouldn't have to.
* `Client` detects whether or not it is connected to a `LocalCluster` and sets the default shuffle config to disk
* The detection should be based on detecting specifically LocalCluster, not on IP ranges or other means
* Deprecation should be announced via documentation or a proper warning
* Benchmark should exist verifying this is faster
Note:
* This could be resolved at graph construction time when using HLG or HLE since the graph materialization is delayed.
Contributor guide
Assessment
This issue has not been assessed yet.