dask / dask/distributed

Out of memory / memory leak debugging

Open
#1,866 14 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Hi,

This is less of a clear bug report and more a writeup of some debugging I recently did around weird memory leak like issues while running code using dask. I hope this will save someone a bit of time in the future.

I'm trying to use dask to run some simple code in parallel, as a better `multiprocessing`. Roughly this:
```python
def fun(x):
return x + 1
tasks = [delayed(fun)(i) for i in range(1000)]
futs = client.compute(tasks)
```

In practice `fun` is a bit more complex, and reads data from s3, does a bit of computation and writes results back to s3.

When running the real example in parallel I was seeing really slow scheduling, and workers slowly ran out of memory. It seemed to scale roughly with the number of tasks, each task takes around 30 seconds on a single core, and when I was running 10 of them it all worked perfectly. 500 still worked, 1000 were borderline and 10000 certainly didn't.

My workers died with various exceptions related to memory usage, e.g.
```
distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting
```

After a bit of debugging this seems to be caused by a helper class that got passed into `fun`, a minimal example that breaks looks like this:
```python
class S3FsWrapper(object):
def __init__(self):
self.fs = s3fs.S3FileSystem()
def get_s3fs(self):
return self.fs

fs = S3FsWrapper()

def fun(fs, x):
# would do something with fs here, but not necessary to trigger OOM
return x + 1

tasks = [delayed(fun)(fs, i) for i in range(1000)]
futs = client.compute(tasks)
progress(futs)
```

Running this will take ages, and depending on how much RAM you have will most likely crash.

So I had a look at where all this memory goes with `pympler.muppy`:
```python
def debug_mem():
from pympler import summary, muppy
all_objects = muppy.get_objects()
s = summary.summarize(all_objects)
return s

s = client.run(debug_mem)

from pympler import summary, muppy
summary.print_(list(s.values())[0])
```
```
types | # objects | total size
======================================================= | =========== | ============

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.