Hanging when using python multiprocessing along with dask.distributed
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
I encountered hanging when using python multiprocessing along with dask.distributed (nested parallelism);
**What you expected to happen**:
I wonder whether that should work.
**Minimal Complete Verifiable Example**:
```python
if __name__ == '__main__':
from distributed import Client
client = Client()
import pandas
df = pandas.DataFrame([1])
fut = client.scatter(df)
def f(obj):
local_df = client.gather(fut)
local_df += obj
from multiprocessing import Pool
with Pool(5) as p:
print(p.map(f, [1]))
```
**Environment**:
- Dask version: dask 2021.11.2 pyhd8ed1ab_0 conda-forge
- Python version: Python 3.8.12
- Operating System: Ubuntu 20.04.2
- Install method (conda, pip, source): conda
Contributor guide
Assessment
This issue has not been assessed yet.