[Parquet] Adaptively to pick between RowSelection and Mask filter representation
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
After the great work from @hhhizzz in https://github.com/apache/arrow-rs/pull/8733, we will (finally) have the ability to use a Bitmask filter representation when applying filters *during* Parquet decode.
At the moment, the code relies on a simple threshold strategy to pick between representations
https://github.com/apache/arrow-rs/blob/911331aafa13f5e230440cf5d02feb245985c64e/parquet/src/arrow/arrow_reader/read_plan.rs#L107-L130
However, as @hhhizzz mentions in https://github.com/apache/arrow-rs/pull/8733#discussion_r2506343981
> Yes, my charts indicate that there are many rules for setting the RowSelectionStrategy, like the column type, column count, string length, and their combinations... We can create tickets and collaborate on improving these over time.
**Describe the solution you'd like**
I would like better heuristics for selecting between the stratgies
**Describe alternatives you've considered**
@hhhizzz has some good suggestions, and the charts from https://github.com/apache/arrow-rs/pull/8733#issuecomment-3468441165 offer some good ideas:
> For how I get the the average length to use the mask, here's some statistic, you can checkout to (https://github.com/hhhizzz/arrow-rs/tree/rowselectionempty-charts) and run `python3 dev/row_selection_analysis.py` on your local machine, this is the results on my x86 PC:
> # One column `int32`, different distribution type:
>
>
>
>
>
> # Different column type:
>
>
>
>
> # Different column counts:
>
>
>
>
>
**Additional context**
Contributor guide
Research direction
Start in parquet/src/arrow/arrow_reader/read_plan.rs at lines 107-130, then read PR 8733 and its discussion about RowSelectionStrategy. Run python3 dev/row_selection_analysis.py to compare the available charts and identify useful inputs such as column type, column count, and string length. Done means the reader chooses RowSelection or mask representation using documented heuristics supported by the analysis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100