apache / apache/arrow-rs

[Parquet] Support page level cache for reading

Open
#8,246 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Previously in parquet, we should read a whold RowGroup into memory and then extract what we need. This is obviously wasted.
Therefore, I thought of to only read the page we need, and cache the pages for future read.
The previous part is solved thanks to #7850 , and I begin to work after this pr released.

**Describe the solution you'd like**
I thought of adding a cache mechanism into `decode_page` in `impl RowGroupReader for SerializedRowGroupReader`. In this way we can avoid some decode and decompress cost.

**Describe alternatives you've considered**
I have considered to also add cache to filter stage, but this part is already implemented.
I have also considered about page level prefetch, but I think it may be not so profitable.

Contributor guide

Open the contributing guide

Research direction

Start by reading `decode_page` in `SerializedRowGroupReader` and the changes from #7850 to understand page-level reading. Compare the existing filter-stage cache, then determine how page caching should avoid repeated decoding and decompression; the work is done when repeated reads reuse cached pages without reading the whole RowGroup.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.