dask / dask/distributed

Possible memory leak when using LocalCluster

Open
#5,960 33 comments 0 reactions 0 assignees View on GitHub
bug memory needs info
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
Memory usage of code using `da.from_array` and `compute` in a for loop grows over time when using a `LocalCluster`.

**What you expected to happen**:
Memory usage should be approximately stable (subject to the GC).

**Minimal Complete Verifiable Example**:

```python
import numpy as np
import dask.array as da
from dask.distributed import Client, LocalCluster

def f(x):
return np.zeros(x.shape, dtype=x.dtype)

def wrapper(x):
if not isinstance(x, da.Array):
x = da.from_array(x, chunks=(1, -1, -1))

return da.blockwise(f, ('nx', 'ny', 'nz'),
x, ('nx', 'ny', 'nz')).compute()

if __name__=='__main__':

cluster = LocalCluster(
processes=True,
n_workers=4,
threads_per_worker=1
)
client = Client(cluster)

nx, ny, nz = 4, 512, 512

for i in range(500):
x = np.random.randn(nx, ny, nz)
wrapper(x)
```

**Anything else we need to know?**:

The output of `mprof run --multiprocess --include-children reproducer.py` visualised by `mprof plot` using a `LocalCluster`:
![localcluster](https://user-images.githubusercontent.com/6582745/159010948-b979b78c-1195-4d87-9b74-f7206c94470d.png)

The output of `mprof run --multiprocess --include-children reproducer.py` visualised by `mprof plot` using the `threads` scheduler:
![threads](https://user-images.githubusercontent.com/6582745/159011048-4f442203-cf8b-430c-8640-51bbc9a1c7c5.png)

The output of `mprof run --multiprocess --include-children reproducer.py` visualised by `mprof plot` using a `LocalCluster` but instantiating `x` as `da.random.standard_normal((nx, ny, nz), chunks=(1, -1, -1))`:
![nofromarray](https://user-images.githubusercontent.com/6582745/159011555-9d897b05-0970-48d9-9dfc-52cfd9b0b0cd.png)

Note that child 0 should be ignored in the `LocalCluster` plots as it is an artifact of the profiling procedure. Memory usage is in fact still climbing in the last plot, but it is much slower than in the first case (using `da.from_array`). My current best guess is that either the `Client` or the `Cluster` is somehow maintaining a reference to an object which should otherwise have been GCed.

**Environment**:

- Dask version: 2022.2.1
- Python version: 3.8.10
- Operating System: Ubuntu 20.04
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.