apache / apache/datafusion

[Optimizer] Consolidate repeated filter-rebuild patterns in PushDownFilter

Open
#22,056 2 comments 1 reaction 1 assignee Claimed by @SubhamSinghal View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Summary
`PushDownFilter` rebuilds filter nodes in many branch-specific paths. This duplication increases maintenance cost and makes rule invariants harder to keep consistent.

## Background and Motivation
#21667 introduces work to centralize filter construction, but push/keep/reinsert logic remains duplicated across plan variants.

Today, each branch tends to repeat the same shape:
1. split predicates into pushable/keep sets
2. rebuild one or more child filters
3. reinsert kept predicates above
4. return transformed plan

This pattern appears in `Sort`, `Distinct`, `Repartition`, `Projection`, `Union`, `Extension`, `Aggregate`, `Window`, `Unnest`, and join-related paths.

## Problem Statement
Filter reconstruction mechanics are duplicated across many branches in `datafusion/optimizer/src/push_down_filter.rs`.

Concrete symptoms:
- repeated `make_filter(...)` and `Arc::new(...)` call patterns
- repeated split/push/keep control flow
- repeated branch-local child replacement logic

## Why This Matters
- Correctness risk: duplicated rewrite logic makes subtle behavior divergence more likely across plan-node branches.
- Invariant drift: filter reconstruction/reinsertion conventions are harder to enforce when spread across many sites.
- Review burden: future changes require auditing many branches for equivalent behavior.
- Evolvability: adding new single-input node rewrites repeats boilerplate and invites copy-paste defects.

## Proposed Direction
Introduce a small, internal unary-node helper for filter reconstruction within `PushDownFilter`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.