apache / apache/arrow

[C++] ParquetFileFragment caches parquet file metadata and there is no way to disable this

Open
#31,824 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

When looking at ARROW-15081 there was a strange amount of memory used even when we were accumulating all of the results into a single 64 byte counter (e.g. `SELECT COUNT(*) FROM table`).

It turns out this was the parquet metadata, which gets attached to the parquet file fragment. There is no way to prevent this and, in this case, it was using quite a bit of RAM. There were 1100 files and each file had ~10MB of metadata.

We should have an option for disabling this. Also, this should probably be off by default. It can be a useful thing to cache if you are going to run the same dataset again and again but otherwise it is just wasted RAM.

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-16451) / @westonpace

**Note**: *This issue was originally created as [ARROW-16451](https://issues.apache.org/jira/browse/ARROW-16451). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with ParquetFileFragment and the metadata behavior described in ARROW-15081. Trace where parquet file metadata is attached and review the two existing comments for context. Done means the behavior can be disabled through an option and the intended default is documented or tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.