dask / dask/distributed

Hanging when using python multiprocessing along with dask.distributed

Open
#5,596 8 comments 0 reactions 0 assignees View on GitHub
needs info
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:

I encountered hanging when using python multiprocessing along with dask.distributed (nested parallelism);

**What you expected to happen**:

I wonder whether that should work.

**Minimal Complete Verifiable Example**:

```python
if __name__ == '__main__':
from distributed import Client
client = Client()

import pandas
df = pandas.DataFrame([1])
fut = client.scatter(df)

def f(obj):
local_df = client.gather(fut)
local_df += obj

from multiprocessing import Pool

with Pool(5) as p:
print(p.map(f, [1]))
```

**Environment**:

- Dask version: dask 2021.11.2 pyhd8ed1ab_0 conda-forge
- Python version: Python 3.8.12
- Operating System: Ubuntu 20.04.2
- Install method (conda, pip, source): conda

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.