dask / dask/distributed

Round-robin worker selection makes poor choices with `worker-saturation > 1.0`

Open
#7,197 6 comments 1 reaction 0 assignees View on GitHub
bug scheduling
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

`test_wait_first_completed` is failing in https://github.com/dask/distributed/pull/7191, with the `worker-saturation` value set to 1.1
https://github.com/dask/distributed/blob/09837312ccb3a9f1a14ea068ba5825963fec82cc/distributed/tests/test_client.py#L732-L746

It works fine with 1.0, but because of the round-up logic https://github.com/dask/distributed/pull/7116 allowing workers to be oversaturated, fails for 1.1

It blocks forever because the worker with 1 thread gets assigned `[block_on_event, inc]`, and the worker with 2 threads gets assigned `[block_on_event]`. It should be the other way around.

The culprit has something to do with the round-robin logic that only applies to rare situations like this, where the cluster is small but larger than the TaskGroup being assigned

https://github.com/dask/distributed/blob/09837312ccb3a9f1a14ea068ba5825963fec82cc/distributed/scheduler.py#L2210-L2236

If I update `is_rootish` like so:

```diff
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
index cf240240..802df12d 100644
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -3043,6 +3043,8 @@ class SchedulerState:
"""
if ts.resource_restrictions or ts.worker_restrictions or ts.host_restrictions:
return False
+ if not ts.dependencies:
+ return True
tg = ts.group
# TODO short-circuit to True if `not ts.dependencies`?
return (
```

the test passes.

cc @fjetter @crusaderky

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.