dask / dask/distributed

Imbalanced scheduling of non-root tasks with resources

Open
#8,177 4 comments 0 reactions 1 assignee Claimed by @fjetter View on GitHub
scheduling
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

From https://dask.discourse.group/t/only-1-worker-is-running-when-the-dag-is-forking/2192

Non-root tasks that declare resources do not evenly distribute on the cluster, instead piling up on a single worker.

```python
import time
import dask
from distributed import Client

@dask.delayed
def f():
return 1

@dask.delayed
def g(x, y):
time.sleep(2)
return x + y

ops = []
root = f()
#root = 1

for i in range(4):
with dask.annotate(resources={"cores": 100}):
nonroot = g(root, i)
ops.append(nonroot)

with Client(n_workers=2, threads_per_worker=4, resources={"cores": 128}):
t0 = time.time()
dask.compute(*ops)
t1 = time.time()

print("compute time:", t1 - t0)
```
Expected: 4s
Actual: 8s

`distributed.scheduler.worker-saturation` does not seem to make a difference. Having less or more than 5 tasks (the threshold for `is_rootish`) doesn't seem to have an impact (as long as you have less tasks than threads).
Uncommenting `root = 1`, thus making the tasks with resources actually root (not just rootish) makes the issue disappear.

After increasing the number of tasks from 6 to 100, this is what I see on the dashboard:
![image](https://github.com/dask/distributed/assets/6213168/41161b46-cb26-426a-8619-e6615a3bcdc1)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.