apache / apache/arrow

[Python] Memory leak with dataset.head

Open
#36,754 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

dataset.head loads all data in memory and doesn't release it. when it should just load the top n rows.

This issue started after July 17 2023.

## Versions
Pyarrow : 12.0.0
Python: 3.10.6
Jupyter lab: 3.3.4 on
Docker: 4.12.0 (85629) on
windows 10, version: 21H2, build: 19044.3086

## Sample data

https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2009-01.parquet For all the months

## Sample Code

1. Install memory_profiler
```
pip3 install memory_profiler
```
2. load Extension and Check mem
```
%load_ext memory_profiler
%memit
```
peak memory: 163.00 MiB, increment: 0.21 MiB

3. Create Dataset
```
import pyarrow.dataset as ds
data = ds.dataset('./testdata/nyc/year=2009', format='parquet', partitioning='hive')
```
4. Check mem
```
%memit
```

peak memory: 157.97 MiB, increment: 0.01 MiB

5. Count rows
```
data.count_rows()
```
170896055

6. Check mem
```
%memit
```
peak memory: 170.34 MiB, increment: 0.02 MiB

7. get first 10 rows
```
data.head(10).to_pandas()
```
8. Check Mem
```
%memit
```

peak memory: 11753.76 MiB, increment: 142.51 MiB
peak memory: 9914.21 MiB, increment: 0.00 MiB
peak memory: 9914.21 MiB, increment: 0.00 MiB
peak memory: 9914.21 MiB, increment: 0.00 MiB

### Component(s)

Parquet, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.