apache / apache/arrow

[C++][Parquet] Loading big parquet files leads to huge memory consumption

Open
#44,890 6 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Loading big parquet files (23GB file) leads to high memory consumption, ending in OOM. Memory consumption surpass my machine limit of 64GB ram. Happens on Linux, arrow 17.0.0, Ubuntu 22

Reader is set up similar as demonstrated on docs page via `` parquet::arrow::FileReaderBuilder`` with key difference of
passing custom allocator. Huge usage of memory is not report by my custom allocator nor by memory pool passed to reader.

After invastigation iI encouter thath RangeReadeCache created lot of cache entries, 735 to be exact witch each about 33MiB of length. Each with it's future with use count 1.
![enteries](https://github.com/user-attachments/assets/e2eb65ab-8163-4bfd-b4cd-fc3dd2335f11)
Also for some reason this uses default memory pool
![image](https://github.com/user-attachments/assets/95b78844-e590-4e67-8c89-45c0d440516b)

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.