[Parquet] Support page level cache for reading
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Previously in parquet, we should read a whold RowGroup into memory and then extract what we need. This is obviously wasted.
Therefore, I thought of to only read the page we need, and cache the pages for future read.
The previous part is solved thanks to #7850 , and I begin to work after this pr released.
**Describe the solution you'd like**
I thought of adding a cache mechanism into `decode_page` in `impl RowGroupReader for SerializedRowGroupReader`. In this way we can avoid some decode and decompress cost.
**Describe alternatives you've considered**
I have considered to also add cache to filter stage, but this part is already implemented.
I have also considered about page level prefetch, but I think it may be not so profitable.
Contributor guide
Research direction
Start by reading `decode_page` in `SerializedRowGroupReader` and the changes from #7850 to understand page-level reading. Compare the existing filter-stage cache, then determine how page caching should avoid repeated decoding and decompression; the work is done when repeated reads reuse cached pages without reading the whole RowGroup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100