apache / apache/arrow-rs

Avoid `RowSelections` in `ReadPlanBuilder`

Open
#9,416 4 comments 1 reaction 0 assignees View on GitHub
performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We can create mask based row selections for parquet filters to avoid the rowselection overhead.
However, before converting them, `RowSelection::from_filters` is called to convert them in `ReadPlanBuilder::with_predicate`, storing them as indices and using `and_then` to combine indices.
In very effective filters, this is not a problem, but it can generate some overhead in non-effective filters.

**Describe the solution you'd like**

Optimize the `ReadPlanBuilder` to:
* store the current selection as `RowSelectionCursor` (or as mask?)
* leave the boolean masks as is for non-effective filters
* use bitwise `and` to combine masks when possible (or efficiently combine `RowSelection` and mask)

**Describe alternatives you've considered**
Keep it as is, optimize `RowSelection` code (from_filters / and_then...)

**Additional context**
Might be necessary to remove all regressions: https://github.com/apache/datafusion/issues/20324

Contributor guide

Open the contributing guide

Research direction

Start with ReadPlanBuilder::with_predicate and trace how RowSelection::from_filters and and_then convert and combine selections. Compare retaining masks with RowSelectionCursor or mixed mask/RowSelection handling, then verify filter behavior and check the regression context in issue 20324.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.