dask / dask/distributed

When using a local cluster, shuffle with disk

Open
#5,502 8 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

In feedback for https://github.com/dask/dask/pull/8223, I've noticed users trying to do large shuffles on single-machine clusters: https://github.com/dask/dask/pull/8223#issuecomment-961610095, https://github.com/dask/dask/issues/8294#issuecomment-961301624.

When creating any default Client, the default shuffle mode automatically gets set to `"tasks"`:
https://github.com/dask/distributed/blob/69814b4aa7459476dcefa133341b566a5ed4e24a/distributed/client.py#L727-L730

However, on a single machine, a disk-based shuffle is likely to be a lot more efficient, plus much lower load on the scheduler.

I think it would be better to keep using the disk-based shuffle if the Client is connected to a LocalCluster (not sure how to tell this). Most users don't know about the different shuffle modes, and shouldn't have to.

* `Client` detects whether or not it is connected to a `LocalCluster` and sets the default shuffle config to disk
* The detection should be based on detecting specifically LocalCluster, not on IP ranges or other means
* Deprecation should be announced via documentation or a proper warning
* Benchmark should exist verifying this is faster

Note:
* This could be resolved at graph construction time when using HLG or HLE since the graph materialization is delayed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.