apache / apache/arrow-rs

Parquet: support same-projection `RowFilter` fusion

Open
#10,926 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Is your feature request related to a problem or challenge?

`RowFilter` evaluates each `ArrowPredicate` separately, so same-projection predicates may decode or replay the same column more than once. #10859 reuses one decoded stream and evaluates later predicates only on surviving rows.

This can avoid repeated decoding or predicate-cache replay, but fragmented or dense survivors may be expensive to materialize. Fusion can therefore regress some workloads.

### Describe the solution you'd like

When consecutive predicates use the same projection, the Parquet reader should be able to evaluate them from one decoded stream. This avoids repeated decoding or predicate-cache replay while preserving predicate order and ensuring that each predicate sees only rows accepted by earlier predicates.

Fusion should not be an unconditional rewrite: as described above, materializing survivors can cost more than the repeated work it removes for some execution shapes. How to decide when fusion applies — a conservative structural rule, caller configuration, or something smarter — is left open for discussion. #10859 currently uses a conservative structural gate (a single top-level, non-repeated leaf projection, and no active row limit).

### Describe alternatives you've considered

_No response_

### Additional context

- Arrow: #10859, #10774, #10776
- DataFusion: apache/datafusion#24426, apache/datafusion#24705

Contributor guide

Open the contributing guide

Research direction

Start with the Parquet reader's RowFilter and ArrowPredicate handling, then read #10859, #10774, and #10776 to understand the existing decoded-stream reuse and structural gate. Define a safe fusion decision that preserves predicate order and survivor semantics, and verify that it avoids repeated decoding without regressing workloads where materializing survivors costs more.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.