Timed out trying to connect when using multiprocessing library
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
It seems that `distributed` does not mesh well with the python's `multiprocessing` library:
```python
from distributed import Client
from multiprocessing import Process
from dask import bag as db
import os
os.environ["OBJC_DISABLE_INITIALIZE_FORK_SAFETY"] = "YES"
def handler():
print("Let's connect to Dask")
c = Client("tcp://127.0.0.1:8687")
print(c)
c.compute(db.from_sequence(range(100000)).map(lambda x: x), sync=True, progress=True)
print("everything got fine")
def mpdask():
"""test if multiprocessing breaks distributed dask"""
print("connecting")
c = Client(scheduler_port=8687, host="127.0.0.1")
print(f"connected to {c}")
p = Process(target=handler)
p.start()
p.join()
if __name__ == '__main__':
mpdask()
```
(the ENV part is required only for some version of macOS)
It gives me:
```
OSError: Timed out trying to connect to 'tcp://127.0.0.1:8687' after 10 s: Timed out trying to connect to 'tcp://127.0.0.1:8687' after 10 s: connect() didn't finish in time
```
in `distributed/comm/core.py` (for every distributed version I try the code)
I can't follow very well how the async connection mechanism is meant to work...
Contributor guide
Assessment
This issue has not been assessed yet.