[C++] ParquetFileFragment caches parquet file metadata and there is no way to disable this
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
When looking at ARROW-15081 there was a strange amount of memory used even when we were accumulating all of the results into a single 64 byte counter (e.g. `SELECT COUNT(*) FROM table`).
It turns out this was the parquet metadata, which gets attached to the parquet file fragment. There is no way to prevent this and, in this case, it was using quite a bit of RAM. There were 1100 files and each file had ~10MB of metadata.
We should have an option for disabling this. Also, this should probably be off by default. It can be a useful thing to cache if you are going to run the same dataset again and again but otherwise it is just wasted RAM.
**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-16451) / @westonpace
**Note**: *This issue was originally created as [ARROW-16451](https://issues.apache.org/jira/browse/ARROW-16451). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with ParquetFileFragment and the metadata behavior described in ARROW-15081. Trace where parquet file metadata is attached and review the two existing comments for context. Done means the behavior can be disabled through an option and the intended default is documented or tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100