dask / dask/distributed

Unmanaged memory leads to program hanging

Open
#7,864 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I am trying to use dask.DataFrame to do a very simple task, the code looks like this.

```
import dask.dataframe as dd
from distributed import Client

client = Client(n_workers= 1,
threads_per_worker= 1,
memory_limit = "12GB")

# load pandas.DataFrame from database.
tmp_data = {name: load_from_database(name) for name in names}

# tmp_data is about 20GB in total, and there are 308 elements.
# convert tmp_data to dask.DataFrame
dfs = [df.from_pandas(df, npartitions=1).repartition(partition_size="100MB") for k, df in tmp_data.items() if df.shape[0] > 0]
data = dd.concat(dfs)

mu_mb = self._data.memory_usage_per_partition(deep=True).compute().sort_values() / 1e6

```

`

The program would run for several seconds and hang, and keeps getting
" warning: distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS. "

The dashboard looks like this
![image](https://github.com/dask/distributed/assets/39135113/e571e456-ff32-410e-b547-7c126899b011)

I tried to set MALLOC_TRIM_THRESHOLD_=0 in the console, before running the code, but it does not seem to work.
Did I do anything wrong? Please help

**Environment**:

- Dask version: 2022.11.1
- Python version: 3.8.10
- Operating System: Ubuntu 20. Total RAM: 64GB
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.