Avoid `RowSelections` in `ReadPlanBuilder`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We can create mask based row selections for parquet filters to avoid the rowselection overhead.
However, before converting them, `RowSelection::from_filters` is called to convert them in `ReadPlanBuilder::with_predicate`, storing them as indices and using `and_then` to combine indices.
In very effective filters, this is not a problem, but it can generate some overhead in non-effective filters.
**Describe the solution you'd like**
Optimize the `ReadPlanBuilder` to:
* store the current selection as `RowSelectionCursor` (or as mask?)
* leave the boolean masks as is for non-effective filters
* use bitwise `and` to combine masks when possible (or efficiently combine `RowSelection` and mask)
**Describe alternatives you've considered**
Keep it as is, optimize `RowSelection` code (from_filters / and_then...)
**Additional context**
Might be necessary to remove all regressions: https://github.com/apache/datafusion/issues/20324
Contributor guide
Research direction
Start with ReadPlanBuilder::with_predicate and trace how RowSelection::from_filters and and_then convert and combine selections. Compare retaining masks with RowSelectionCursor or mixed mask/RowSelection handling, then verify filter behavior and check the regression context in issue 20324.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100