apache / apache/arrow

Memory leak on Windows when reading parquet with mixed dtypes via Pyarrow

Open
#38,736 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I've been noticing a memory leak for several years now. When reading a big parquet file, pyarrow lib or Pandas with pyarrow engine (default one) suffer from extensive RAM waste, twice the size of data itself. RAM is not freed by gc.collect(), even after the dataframe in question has been deleted from the workspace (you can try uncommenting deletion lines in my file).

Finally found time to reproduce and report it.

Let's first create a big file with lots of columns and mixed dtypes:

test_parquet_make_file.py

```python
import pyarrow.parquet as pq
import pandas as pd, numpy as np

df = pd.DataFrame(np.random.random(size=(100_000,10000)))
for i in [10,100,1000,5000]:
df.iloc[:,i]=df.iloc[:,i].astype(np.int32)
df.iloc[:,i+1]=df.iloc[:,i+1].astype(np.float32)
df['cat']='cat'
df['cat']=df['cat'].astype('category')
print(df.info())
df.to_parquet("output.parquet")
```
It gets saved as an approx. 10Gb file.

Let's now open it with pyarrow, report its size, wait a bit, garbage collect, and report process RAM usage:

test_parquet_leak.py

```python
import pyarrow.parquet as pq
from gc import collect
from time import sleep
import pandas as pd, numpy as np
import psutil, os

#df = pd.read_parquet("output.parquet",engine='fastparquet')
#df = pd.read_parquet("output.parquet",engine='pyarrow')
#print(df.info())

df=pq.read_table("output.parquet")

collect()
sleep(5)
collect()

print(f"Data size: {df.get_total_buffer_size()/1024**3:.2f} Gb")

#del df
#collect()
#sleep(5)
#collect()

process = psutil.Process(os.getpid())
try:
mem=process.memory_info()
except:
mem=process.full_memory_info()
print(f"process RAM used: {mem.rss / 1024**3:.2f} Gb")
```

Output on my Windows x64 station:

>
> Data size: 7.57 Gb
> process RAM used: 21.90 Gb

By uncommenting the lines, you can see that Pandas with Pyarrow engine also suffers from this, **unlike fastparquet engine**.
Again, even if data object is deleted, memory stays taken in the process forever.
For big files, this creates unnecesary OOM problems. I can understand temporarily usage of more RAM for performance reasons, but IMHO there should be no excuse for permanent RAM lock up like this in such significant amounts (2-x/3x of original data).
Thanks!

### Component(s)

Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.