dask / dask/distributed

Huge memory leak and processes do not restart automatically

Open
#4,345 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I have very long running task with thousands of tasks being send to the workers.
Since there are known problems related to memory leakage, at some point the RAM is full and at that point the `dask` workers are still alive but idle, they are stuck.

Looking into the source code I see that, since I use `processes=True` the workers should be automatically created as `Nanny` processes and there is a parameter called `auto_restart=True` meaning that when the process reaches its memory limit it should be restarted automatically.
However this is not happening as I explained above.

Any idea of why? (I'll try to work on a minimal reproducible example asap)
Many thanks
Gio

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.