apache / apache/datafusion

[Follow-up #23696] Prebuild parquet row-filter candidates once per file

Open
#24,563 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Follow-up to #23696, per @adriangb's [decomposition proposal](https://github.com/apache/datafusion/pull/23696#issuecomment-5346992228) (PR 2).

`RowFilterGenerator::build()` currently redoes `split_conjunction` + `FilterCandidateBuilder::build` + `reassign_expr_columns` **for every row group**. On any multi-row-group scan with a pushdown predicate, that per-RG tree walk and column resolution is wasted work.

Move it to **once per file**: prebuild the row-filter candidate list when the file is opened, then instantiate a `RowFilter` per row group cheaply from the prebuilt list.

Proposed shape (extracted from #23696, decoupled from `fully_matched`):
- add `PrebuiltRowFilterCandidate` / `prebuild_row_filter_candidates` (once per file: tree walk + candidate construction) and `row_filter_from_prebuilt` (per RG: cheap bind against the fresh array readers);
- delete `RowFilterGenerator`;
- **reimplement the existing public `build_row_filter` on top of the two new functions** (its signature must stay — `parquet_nested_filter_pushdown` / `parquet_struct_filter_pushdown` benches depend on it).

Parity to preserve: same conjunct order (`sort_unstable_by_key(required_bytes)` when `reorder_predicates`), same metric wiring (every predicate shares `pushdown_rows_pruned`, only the last predicate counts `pushdown_rows_matched`). One intended behavioural delta: `reassign_expr_columns` errors surface once at open time instead of being swallowed per-build by `log::debug!`.

Needs nothing from `fully_matched`.

Contributor guide

Open the contributing guide

Research direction

Start by tracing RowFilterGenerator::build and the public build_row_filter entry point, then inspect the parquet_nested_filter_pushdown and parquet_struct_filter_pushdown benchmarks. Done means candidate construction occurs once per file, row-group filters are instantiated cheaply, the public signature remains unchanged, and predicate ordering and metric wiring retain parity.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.