apache / apache/datafusion

Dynamic filter no-op for NULL-only build keys (NullEqualsNothing)

Open
#17,206 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Summary
-------
When the build side of a hash join contains only NULL join keys and the join is configured with
`datafusion_common::NullEquality::NullEqualsNothing`, the dynamic filter generated for the probe side
is produced as range comparisons against `NULL` (e.g. `a >= NULL AND a <= NULL`). In the current
optimizer/test code this is treated as a no-op (a tautology) instead of being treated as either
unsatisfiable or as an explicit "no matches" condition.

Why this matters
----------------
- This is a surprising corner case and may hide regressions if semantics or simplification rules change.
- If NullEquality semantics change (or the filter simplifier is updated), behavior and test expectations
could silently diverge.
- We should monitor and decide whether this should be:
- left as a tautology (no-op),
- treated as unsatisfiable (prune everything),
- or canonicalized/annotated to make intent explicit.

Where to look / repro
---------------------
Test: `datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs`
Specifically the test `test_hashjoin_dynamic_filter_pushdown_null_keys` in #17090

Reproduction steps (test harness)
1. Run that single async test (or the optimizer tests):
- cargo test --test --filter test_hashjoin_dynamic_filter_pushdown_null_keys
2. Observe plan printed by `format_plan_for_test(&plan)` contains:
`DynamicFilterPhysicalExpr [ a@0 >= NULL AND a@0 <= NULL AND b@1 >= NULL AND b@1 <= NULL ]`

Observed behavior
-----------------
The optimizer generates a dynamic filter with min/max bounds set to NULL; the code interprets this as
a tautology (no filtering). The current test documents this and asserts the presence of the `>= NULL` / `<= NULL` pattern.

Expected / options
------------------
We need a decision on intended semantics. Options:
- Keep current behavior (tautology/no-op). Document it clearly in code/tests.
- Treat NULL-only build-side as unsatisfiable filter (drop all probe rows).
- Change filter generation to avoid producing `>= NULL` / `<= NULL` and instead produce an explicit marker (e.g., `unsatisfiable`) so optimizer simplification can handle it deterministically.

References
----------
File: `datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs`
Test: `test_hashjoin_dynamic_filter_pushdown_null_keys`

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs and the test test_hashjoin_dynamic_filter_pushdown_null_keys. Run the single async test or the optimizer tests and inspect the formatted plan containing comparisons against NULL. A decision is needed on whether this remains a tautology, becomes unsatisfiable, or uses an explicit marker; done means the chosen semantics are reflected consistently in the implementation and tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.