dask / dask/distributed

Avoid load balancing when there is plenty of load

Open
#5,243 0 comments 0 reactions 0 assignees View on GitHub
stealing
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm looking at a computation right now where the scheduler is engaging work stealing between workers that have more work to workers that have less work. This makes a lot of sense, except for the fact that all of the workers have a ton of work.

We probably should care less about balancing occupancy as the ratio of occupancy to average task duration is high. It's going to be a while before we need to care about load balancing, and things may change by the time that that happens. Right now small variances in occupancy (say, 10%) are causing the cluster to move around a lot of data, which is probably unnecessary.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.