apache / apache/arrow

High (resident) memory usage when fetching Parquet metadata/schema

Open
#35,393 18 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

We have a code to fetch parquet schema from a file using pyarrow, here's a minimal example:

```py
import pyarrow.parquet as pq

with open("/tmp/part.snappy.parquet", mode="rb") as fd:
s = pq.read_schema(fd)
```

That example file is about 288MB, we've notice that the resident memory usage of this code spikes close to 500MB:

image

Is this expected that to fetch schema, we need to allocate so much memory? Worth noting that this memory is eventually freed up. Should some arguments be tweaked or is this a bug somewhere?

```sh
> du -sh /tmp/part.snappy.parquet
288M /tmp/part.snappy.parquet
```

Versions (py 3.10):
```
> conda list | grep arrow
arrow-cpp 12.0.0 hce30654_0_cpu conda-forge
libarrow 12.0.0 h3b4cbd9_0_cpu conda-forge
pyarrow 12.0.0 py310h7c67832_0_cpu conda-forge
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the minimal pyarrow.parquet.read_schema example with the supplied 288MB file and the listed Arrow 12.0.0/Python 3.10 versions, while measuring resident memory. Compare the allocation and release behavior with the expected metadata-read path; done means determining whether the spike is expected or identifying a reproducible defect and its relevant component.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.