Join planning inserts IS NOT NULL filters that provably pass 100% of rows when input statistics report zero nulls
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Summary
Join planning inserts null-rejecting `IS NOT NULL` filters on nullable join keys (`filter_null_join_keys`), which are then pushed down to scans. When the underlying data contains no NULLs — the overwhelmingly common case for join keys, e.g. TPC-H primary/foreign keys — these filters pass 100% of rows, yet every input batch still pays a full predicate evaluation plus a full batch copy.
On TPC-H SF=10 we measured `selectivity = 100% (6.00M/6.00M)` for such filters on join keys.
## Why this is wasteful
`FilterExec::statistics_helper` already derives the forward direction: a surviving `IS NOT NULL` conjunct implies `null_count = Exact(0)` for the output. The inverse is missing: when *input* statistics already prove `null_count = Exact(0)` for a column, a bare `Column IS NOT NULL` conjunct is vacuously true and could be dropped from the predicate at plan-construction time (`FilterExecBuilder::build`). If all conjuncts are dropped, the predicate degenerates to `lit(true)` and the filter becomes a no-op pass-through (or can be elided entirely).
Non-provable cases (statistics `Absent` or null_count `Inexact`) must of course keep the predicate unchanged — this is strictly a statistics-driven refinement.
## Impact
- Every join query over sources that report exact zero null-counts (Parquet with statistics, custom TableProviders with exact stats) executes no-op filter evaluations per batch.
- For plans with many joins (TPC-H), the redundant filters stack up on the scan side.
## Related
- PR implementing the conjunct-drop at `FilterExecBuilder::build`: #24821 (filed this issue per review discussion there)
Contributor guide
Research direction
Read FilterExecBuilder::build and FilterExec::statistics_helper first, then inspect how input null-count statistics are represented for filter columns. Done means dropping only provably true Column IS NOT NULL conjuncts when null_count is Exact(0), retaining predicates for Absent or Inexact statistics, and making an all-dropped filter a no-op.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100