Potential race condition in Nanny
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Hi everybody, since a few days we're seeing "random" failures in our CI due to `distributed` emitting:
```
tornado.application - ERROR - Exception in callback >
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.7.7/x64/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
return self.callback()
File "/opt/hostedtoolcache/Python/3.7.7/x64/lib/python3.7/site-packages/distributed/nanny.py", line 414, in memory_monitor
process = self.process.process
AttributeError: 'NoneType' object has no attribute 'process'
```
This feels like a race condition in some situation, e.g. closing of the `Nanny` because the periodic callbacks are still running but `Nanny.process` is already None.
I **think** (still investigating) we've started seeing this only after 2.20 was released.
I cannot attach an MFE yet simply because we don't have one 😬 we only experience this on CI. I'd welcome any sort of feedback. I can try to give some context though: the error is triggered in a Jupyter Notebook by a cell which calls `scipy.minimize` from a DASK worker (cell 14 [here](https://github.com/devitocodes/devito/blob/master/examples/seismic/tutorials/04_dask.ipynb) -- look for `optimize.minimize`). I doubt it's ever going to be useful, but [here](https://pastebin.com/VFzadUwr)'s an excerpt of the raw log (part of which I pasted above): note that it repeats over and over again for hundreds/thousands of lines...
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.