LocalCluster startup time
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Starting workers with LocalCluster is slow, and linear in the number of workers, such that those with >100 processes have to wait a significant time for the client creation to return. Laptop users probably don't notice this at all, although the typical number of cores is higher than it used to be.
Example
```
In [1]: import distributed
In [2]: %time client = distributed.Client(n_workers=1) # warmup
CPU times: user 369 ms, sys: 91.4 ms, total: 461 ms
Wall time: 1.46 s
In [3]: client.close()
In [4]: %time client = distributed.Client(n_workers=1)
CPU times: user 33.2 ms, sys: 14.5 ms, total: 47.7 ms
Wall time: 747 ms
In [5]: client.close()
In [6]: %time client = distributed.Client(n_workers=4)
CPU times: user 47.7 ms, sys: 32.4 ms, total: 80.1 ms
Wall time: 871 ms
In [7]: client.close()
In [8]: %time client = distributed.Client(n_workers=10)
CPU times: user 99.1 ms, sys: 73.9 ms, total: 173 ms
Wall time: 1.32 s
In [9]: client.close()
In [10]: %time client = distributed.Client(n_workers=30)
CPU times: user 302 ms, sys: 224 ms, total: 526 ms
Wall time: 3.96 s
In [11]: client.close()
```
Essentially all of this time is spent in Nanny._wait_until_connected , i.e., we only return when every worker comes online - even though we don't explicitly `client.wait_for_workers`. Starting with no workers and then scaling does not help.
Actually launching a process takes of order 10ms on my machine.
Contributor guide
Assessment
This issue has not been assessed yet.