Dynamic filter no-op for NULL-only build keys (NullEqualsNothing)
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Summary
-------
When the build side of a hash join contains only NULL join keys and the join is configured with
`datafusion_common::NullEquality::NullEqualsNothing`, the dynamic filter generated for the probe side
is produced as range comparisons against `NULL` (e.g. `a >= NULL AND a <= NULL`). In the current
optimizer/test code this is treated as a no-op (a tautology) instead of being treated as either
unsatisfiable or as an explicit "no matches" condition.
Why this matters
----------------
- This is a surprising corner case and may hide regressions if semantics or simplification rules change.
- If NullEquality semantics change (or the filter simplifier is updated), behavior and test expectations
could silently diverge.
- We should monitor and decide whether this should be:
- left as a tautology (no-op),
- treated as unsatisfiable (prune everything),
- or canonicalized/annotated to make intent explicit.
Where to look / repro
---------------------
Test: `datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs`
Specifically the test `test_hashjoin_dynamic_filter_pushdown_null_keys` in #17090
Reproduction steps (test harness)
1. Run that single async test (or the optimizer tests):
- cargo test --test --filter test_hashjoin_dynamic_filter_pushdown_null_keys
2. Observe plan printed by `format_plan_for_test(&plan)` contains:
`DynamicFilterPhysicalExpr [ a@0 >= NULL AND a@0 <= NULL AND b@1 >= NULL AND b@1 <= NULL ]`
Observed behavior
-----------------
The optimizer generates a dynamic filter with min/max bounds set to NULL; the code interprets this as
a tautology (no filtering). The current test documents this and asserts the presence of the `>= NULL` / `<= NULL` pattern.
Expected / options
------------------
We need a decision on intended semantics. Options:
- Keep current behavior (tautology/no-op). Document it clearly in code/tests.
- Treat NULL-only build-side as unsatisfiable filter (drop all probe rows).
- Change filter generation to avoid producing `>= NULL` / `<= NULL` and instead produce an explicit marker (e.g., `unsatisfiable`) so optimizer simplification can handle it deterministically.
References
----------
File: `datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs`
Test: `test_hashjoin_dynamic_filter_pushdown_null_keys`
Contributor guide
Research direction
Start with datafusion/core/tests/physical_optimizer/filter_pushdown/mod.rs and the test test_hashjoin_dynamic_filter_pushdown_null_keys. Run the single async test or the optimizer tests and inspect the formatted plan containing comparisons against NULL. A decision is needed on whether this remains a tautology, becomes unsatisfiable, or uses an explicit marker; done means the chosen semantics are reflected consistently in the implementation and tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100