dask / dask/distributed

Timed out trying to connect when using multiprocessing library

Open
#3,290 5 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

It seems that `distributed` does not mesh well with the python's `multiprocessing` library:

```python
from distributed import Client
from multiprocessing import Process
from dask import bag as db

import os
os.environ["OBJC_DISABLE_INITIALIZE_FORK_SAFETY"] = "YES"

def handler():
print("Let's connect to Dask")
c = Client("tcp://127.0.0.1:8687")
print(c)
c.compute(db.from_sequence(range(100000)).map(lambda x: x), sync=True, progress=True)
print("everything got fine")

def mpdask():
"""test if multiprocessing breaks distributed dask"""
print("connecting")
c = Client(scheduler_port=8687, host="127.0.0.1")
print(f"connected to {c}")
p = Process(target=handler)
p.start()
p.join()

if __name__ == '__main__':
mpdask()
```
(the ENV part is required only for some version of macOS)

It gives me:
```
OSError: Timed out trying to connect to 'tcp://127.0.0.1:8687' after 10 s: Timed out trying to connect to 'tcp://127.0.0.1:8687' after 10 s: connect() didn't finish in time
```
in `distributed/comm/core.py` (for every distributed version I try the code)

I can't follow very well how the async connection mechanism is meant to work...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.