Client cannot connect to scheduler when using multiprocessing
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I have a standalone function `f`. I also have a dask scheduler, along with a worker (in separate processes). Because this function should be able to run either in the current process (sync) or in a separate one (async), I infer that it should independently connect to the scheduler, using its own `dask.distributed.Client`.
I first start a scheduler:
```shell
$ dask-scheduler --host localhost
```
then a worker:
```shell
$ dask-worker tcp://localhost:8786
```
then run this code:
```python
from multiprocessing import Process
from dask.distributed import Client
def f():
Client('tcp://localhost:8786', timeout='2s')
print('ok!')
if __name__ == '__main__':
# Async call 1
Process(target=f).start()
# Sync call 2
f() # <-- the presence of this one breaks the next one; when commented out everything is fine
# Async call 3
Process(target=f).start() # <-- will crash because the client will time out!
```
As it is, the client of the third call times out, when trying to connect to the scheduler. If the second, sync call is commented out, everything is fine (only having sync calls is also fine).
Why is this particular combination of sync and async calls causing a problem to the dask client?
Although a bit different in its logic, this issue seems to be closely related to that one: https://github.com/dask/distributed/issues/3290
From there, I seem to understand that using either `multiprocessing.set_start_method("spawn")` (on Linux at least) or `distributed.utils.mp_context` (which appears to be equivalent), mitigates the problem. However, since in my real context it also introduces some additional headaches with pickling, I would be interested in an alternate solution, not involving it.
Contributor guide
Assessment
This issue has not been assessed yet.