apache / apache/datafusion

Join planning inserts IS NOT NULL filters that provably pass 100% of rows when input statistics report zero nulls

Open
#24,942 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Summary

Join planning inserts null-rejecting `IS NOT NULL` filters on nullable join keys (`filter_null_join_keys`), which are then pushed down to scans. When the underlying data contains no NULLs — the overwhelmingly common case for join keys, e.g. TPC-H primary/foreign keys — these filters pass 100% of rows, yet every input batch still pays a full predicate evaluation plus a full batch copy.

On TPC-H SF=10 we measured `selectivity = 100% (6.00M/6.00M)` for such filters on join keys.

## Why this is wasteful

`FilterExec::statistics_helper` already derives the forward direction: a surviving `IS NOT NULL` conjunct implies `null_count = Exact(0)` for the output. The inverse is missing: when *input* statistics already prove `null_count = Exact(0)` for a column, a bare `Column IS NOT NULL` conjunct is vacuously true and could be dropped from the predicate at plan-construction time (`FilterExecBuilder::build`). If all conjuncts are dropped, the predicate degenerates to `lit(true)` and the filter becomes a no-op pass-through (or can be elided entirely).

Non-provable cases (statistics `Absent` or null_count `Inexact`) must of course keep the predicate unchanged — this is strictly a statistics-driven refinement.

## Impact

- Every join query over sources that report exact zero null-counts (Parquet with statistics, custom TableProviders with exact stats) executes no-op filter evaluations per batch.
- For plans with many joins (TPC-H), the redundant filters stack up on the scan side.

## Related

- PR implementing the conjunct-drop at `FilterExecBuilder::build`: #24821 (filed this issue per review discussion there)

Contributor guide

Open the contributing guide

Research direction

Read FilterExecBuilder::build and FilterExec::statistics_helper first, then inspect how input null-count statistics are represented for filter columns. Done means dropping only provably true Column IS NOT NULL conjuncts when null_count is Exact(0), retaining predicates for Absent or Inexact statistics, and making an all-dropped filter a no-op.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.