[Follow-up #23696] Prebuild parquet row-filter candidates once per file
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Follow-up to #23696, per @adriangb's [decomposition proposal](https://github.com/apache/datafusion/pull/23696#issuecomment-5346992228) (PR 2).
`RowFilterGenerator::build()` currently redoes `split_conjunction` + `FilterCandidateBuilder::build` + `reassign_expr_columns` **for every row group**. On any multi-row-group scan with a pushdown predicate, that per-RG tree walk and column resolution is wasted work.
Move it to **once per file**: prebuild the row-filter candidate list when the file is opened, then instantiate a `RowFilter` per row group cheaply from the prebuilt list.
Proposed shape (extracted from #23696, decoupled from `fully_matched`):
- add `PrebuiltRowFilterCandidate` / `prebuild_row_filter_candidates` (once per file: tree walk + candidate construction) and `row_filter_from_prebuilt` (per RG: cheap bind against the fresh array readers);
- delete `RowFilterGenerator`;
- **reimplement the existing public `build_row_filter` on top of the two new functions** (its signature must stay — `parquet_nested_filter_pushdown` / `parquet_struct_filter_pushdown` benches depend on it).
Parity to preserve: same conjunct order (`sort_unstable_by_key(required_bytes)` when `reorder_predicates`), same metric wiring (every predicate shares `pushdown_rows_pruned`, only the last predicate counts `pushdown_rows_matched`). One intended behavioural delta: `reassign_expr_columns` errors surface once at open time instead of being swallowed per-build by `log::debug!`.
Needs nothing from `fully_matched`.
Contributor guide
Research direction
Start by tracing RowFilterGenerator::build and the public build_row_filter entry point, then inspect the parquet_nested_filter_pushdown and parquet_struct_filter_pushdown benchmarks. Done means candidate construction occurs once per file, row-group filters are instantiated cheaply, the public signature remains unchanged, and predicate ordering and metric wiring retain parity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100