more routine garbage collection in distributed?
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I've noticed the memory seems to increase so I was worried of memory leaks. I haven't noticed any (as I'm sure you were very confident I'd say ;-) ).
However, what I have noticed is that sometimes when a python process is killed the memory usage on the cluster doesn't go to zero right away. This can be problematic if the memory usage is quite large.
For example, let's say we have the following code, called `test_distributed.py`
```python
from distributed import Client
client = Client("IP:PORT") # put IP and PORT of sched here
import numpy as np
def foo(a):
return a+1
arr = np.ones((100000000))
arr2 = np.zeros((100000000))
ff = client.submit(foo, arr)
ff2 = client.submit(foo, arr2)
```
If I manually run in 5 times (`python test_distributed.py`), I see the following result for the memory usage:

The memory usage goes up, then comes down when the process terminates, but does not go to zero. When I run the same process, it goes up again, but never exceeds the previous memory usage. So this suggests there is no memory leak.
I figured this could perhaps have something to do with the python garbage collection process, so I went one step further and ran the following script:
```python
from distributed import Client
client = Client("IP:PORT") # put IP and PORT of sched here
def cleanup():
import gc
gc.collect()
client.submit(cleanup)
```
This brought down the memory back to zero.

My feeling is that the python garbage collector can sometimes be slightly more aggressive with memory.
For long running applications like `distributed`, I think it could be a good idea to force the garbage collection process every once in a while.
What do you think? Am I correct in my guess, and would there be a way to resolve this on the `distributed` side? The other obvious solution is for the user to run a cron script sending gc messages to the cluster. However, this is not so clean (and for large intermittent loads may run at very irregular times).
I sort of looked around to see if this was mentioned before, and didn't see anything. I apologize if this is a repost. Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.