dask / dask/distributed

Creating a LocalCluster on a multi-core machine can take a while

Open
#2,450 7 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm playing with a machine that has 80 logical cores. In this case creating a `LocalCluster` or raw `Client` can take a while. I believe that this is because it tries creating 80 workers at once with forkserver.

```python
In [1]: from dask.distributed import Client

In [2]: %time client = Client()
distributed.nanny - WARNING - Worker process still alive after 47 seconds, killing
distributed.nanny - WARNING - Worker process 15200 was killed by unknown signal

```

There are a few potential approaches to solve this:

1. We can try to reduce the cost of creating a new process running a worker. My *guess* is that we're spending a long time importing libraries, but I'm not sure. Someone could investigate this.
2. We can use the multiprocessing `fork` approach rather than forkserver (though dask breaks for other reasons here)
3. We can change the mixture of threads and processes as we get to high number of cores

```python
In [1]: from dask.distributed import Client

In [2]: %time client = Client(n_workers=8, threads_per_worker=10)
CPU times: user 984 ms, sys: 252 ms, total: 1.24 s
Wall time: 6.43 s
```

Probably we should do some combination of 1 and 3. Any thoughts or suggestions?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.