Huge memory leak and processes do not restart automatically
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I have very long running task with thousands of tasks being send to the workers.
Since there are known problems related to memory leakage, at some point the RAM is full and at that point the `dask` workers are still alive but idle, they are stuck.
Looking into the source code I see that, since I use `processes=True` the workers should be automatically created as `Nanny` processes and there is a parameter called `auto_restart=True` meaning that when the process reaches its memory limit it should be restarted automatically.
However this is not happening as I explained above.
Any idea of why? (I'll try to work on a minimal reproducible example asap)
Many thanks
Gio
Contributor guide
Assessment
This issue has not been assessed yet.