dask / dask/distributed

LocalCluster startup time

Open
#8,205 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Starting workers with LocalCluster is slow, and linear in the number of workers, such that those with >100 processes have to wait a significant time for the client creation to return. Laptop users probably don't notice this at all, although the typical number of cores is higher than it used to be.

Example
```
In [1]: import distributed

In [2]: %time client = distributed.Client(n_workers=1) # warmup
CPU times: user 369 ms, sys: 91.4 ms, total: 461 ms
Wall time: 1.46 s

In [3]: client.close()

In [4]: %time client = distributed.Client(n_workers=1)
CPU times: user 33.2 ms, sys: 14.5 ms, total: 47.7 ms
Wall time: 747 ms

In [5]: client.close()

In [6]: %time client = distributed.Client(n_workers=4)
CPU times: user 47.7 ms, sys: 32.4 ms, total: 80.1 ms
Wall time: 871 ms

In [7]: client.close()

In [8]: %time client = distributed.Client(n_workers=10)
CPU times: user 99.1 ms, sys: 73.9 ms, total: 173 ms
Wall time: 1.32 s

In [9]: client.close()

In [10]: %time client = distributed.Client(n_workers=30)
CPU times: user 302 ms, sys: 224 ms, total: 526 ms
Wall time: 3.96 s

In [11]: client.close()
```

Essentially all of this time is spent in Nanny._wait_until_connected , i.e., we only return when every worker comes online - even though we don't explicitly `client.wait_for_workers`. Starting with no workers and then scaling does not help.

Actually launching a process takes of order 10ms on my machine.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.