dask / dask/distributed

LocalCluster not reaping workers with run_on_scheduler and multiprocessing-method: spawn

Open
#1,943 5 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Running the following:

```python
from dask import distributed

def noop():
pass

def noop_on_scheduler(i):
distributed.get_worker().client.run_on_scheduler(noop)

if __name__ == '__main__':
while True:
with distributed.Client(n_workers=8, threads_per_worker=1) as client:
distributed.wait(client.map(noop_on_scheduler, range(8)))
```

I get lots of worker processes being created but not destroyed, and when I hit Ctrl-C, I get lots of warning logs like:

```
distributed.process - WARNING - reaping stray process
```

Versions:

```
- dask=0.17.2=py36_0
- dask-core=0.17.2=py36_0
- distributed=1.21.6=py36_0
- tornado=4.5.3=py36_0
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.