Out of memory / memory leak debugging
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Hi,
This is less of a clear bug report and more a writeup of some debugging I recently did around weird memory leak like issues while running code using dask. I hope this will save someone a bit of time in the future.
I'm trying to use dask to run some simple code in parallel, as a better `multiprocessing`. Roughly this:
```python
def fun(x):
return x + 1
tasks = [delayed(fun)(i) for i in range(1000)]
futs = client.compute(tasks)
```
In practice `fun` is a bit more complex, and reads data from s3, does a bit of computation and writes results back to s3.
When running the real example in parallel I was seeing really slow scheduling, and workers slowly ran out of memory. It seemed to scale roughly with the number of tasks, each task takes around 30 seconds on a single core, and when I was running 10 of them it all worked perfectly. 500 still worked, 1000 were borderline and 10000 certainly didn't.
My workers died with various exceptions related to memory usage, e.g.
```
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
```
After a bit of debugging this seems to be caused by a helper class that got passed into `fun`, a minimal example that breaks looks like this:
```python
class S3FsWrapper(object):
def __init__(self):
self.fs = s3fs.S3FileSystem()
def get_s3fs(self):
return self.fs
fs = S3FsWrapper()
def fun(fs, x):
# would do something with fs here, but not necessary to trigger OOM
return x + 1
tasks = [delayed(fun)(fs, i) for i in range(1000)]
futs = client.compute(tasks)
progress(futs)
```
Running this will take ages, and depending on how much RAM you have will most likely crash.
So I had a look at where all this memory goes with `pympler.muppy`:
```python
def debug_mem():
from pympler import summary, muppy
all_objects = muppy.get_objects()
s = summary.summarize(all_objects)
return s
s = client.run(debug_mem)
from pympler import summary, muppy
summary.print_(list(s.values())[0])
```
```
types | # objects | total size
======================================================= | =========== | ============
Contributor guide
Assessment
This issue has not been assessed yet.