Unmanaged memory leads to program hanging
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
I am trying to use dask.DataFrame to do a very simple task, the code looks like this.
```
import dask.dataframe as dd
from distributed import Client
client = Client(n_workers= 1,
threads_per_worker= 1,
memory_limit = "12GB")
# load pandas.DataFrame from database.
tmp_data = {name: load_from_database(name) for name in names}
# tmp_data is about 20GB in total, and there are 308 elements.
# convert tmp_data to dask.DataFrame
dfs = [df.from_pandas(df, npartitions=1).repartition(partition_size="100MB") for k, df in tmp_data.items() if df.shape[0] > 0]
data = dd.concat(dfs)
mu_mb = self._data.memory_usage_per_partition(deep=True).compute().sort_values() / 1e6
```
`
The program would run for several seconds and hang, and keeps getting
" warning: distributed.worker_memory - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS. "
The dashboard looks like this

I tried to set MALLOC_TRIM_THRESHOLD_=0 in the console, before running the code, but it does not seem to work.
Did I do anything wrong? Please help
**Environment**:
- Dask version: 2022.11.1
- Python version: 3.8.10
- Operating System: Ubuntu 20. Total RAM: 64GB
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.