apache / apache/arrow-rs

[Parquet] Adaptively to pick between RowSelection and Mask filter representation

Open
#8,846 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
After the great work from @hhhizzz in https://github.com/apache/arrow-rs/pull/8733, we will (finally) have the ability to use a Bitmask filter representation when applying filters *during* Parquet decode.

At the moment, the code relies on a simple threshold strategy to pick between representations

https://github.com/apache/arrow-rs/blob/911331aafa13f5e230440cf5d02feb245985c64e/parquet/src/arrow/arrow_reader/read_plan.rs#L107-L130

However, as @hhhizzz mentions in https://github.com/apache/arrow-rs/pull/8733#discussion_r2506343981

> Yes, my charts indicate that there are many rules for setting the RowSelectionStrategy, like the column type, column count, string length, and their combinations... We can create tickets and collaborate on improving these over time.

**Describe the solution you'd like**
I would like better heuristics for selecting between the stratgies

**Describe alternatives you've considered**
@hhhizzz has some good suggestions, and the charts from https://github.com/apache/arrow-rs/pull/8733#issuecomment-3468441165 offer some good ideas:

> For how I get the the average length to use the mask, here's some statistic, you can checkout to (https://github.com/hhhizzz/arrow-rs/tree/rowselectionempty-charts) and run `python3 dev/row_selection_analysis.py` on your local machine, this is the results on my x86 PC:
> # One column `int32`, different distribution type:
> scenario-dense80-dense80
> scenario-sparse20-sparse20
> scenario-spread50-spread50
> scenario-uniform50-uniform50
>
> # Different column type:
> dtype-int32-uniform50
> dtype-utf8view-uniform50
> dtype-float64-uniform50
>
> # Different column counts:
> columns-C02-uniform50
> columns-C04-uniform50
> columns-C08-uniform50
> columns-C16-uniform50
> columns-C32-uniform50

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start in parquet/src/arrow/arrow_reader/read_plan.rs at lines 107-130, then read PR 8733 and its discussion about RowSelectionStrategy. Run python3 dev/row_selection_analysis.py to compare the available charts and identify useful inputs such as column type, column count, and string length. Done means the reader chooses RowSelection or mask representation using documented heuristics supported by the analysis.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.