dask / dask/distributed

Potential race condition in Nanny

Open
#3,955 28 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Hi everybody, since a few days we're seeing "random" failures in our CI due to `distributed` emitting:
```
tornado.application - ERROR - Exception in callback >
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.7.7/x64/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/opt/hostedtoolcache/Python/3.7.7/x64/lib/python3.7/site-packages/distributed/nanny.py", line 414, in memory_monitor
process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'
```

This feels like a race condition in some situation, e.g. closing of the `Nanny` because the periodic callbacks are still running but `Nanny.process` is already None.

I **think** (still investigating) we've started seeing this only after 2.20 was released.

I cannot attach an MFE yet simply because we don't have one 😬 we only experience this on CI. I'd welcome any sort of feedback. I can try to give some context though: the error is triggered in a Jupyter Notebook by a cell which calls `scipy.minimize` from a DASK worker (cell 14 [here](https://github.com/devitocodes/devito/blob/master/examples/seismic/tutorials/04_dask.ipynb) -- look for `optimize.minimize`). I doubt it's ever going to be useful, but [here](https://pastebin.com/VFzadUwr)'s an excerpt of the raw log (part of which I pasted above): note that it repeats over and over again for hundreds/thousands of lines...

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.