apache / apache/datafusion-ballista
Default join planning loses NOT IN null semantics in SortMergeJoinExec
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
## Bug
Ballista defaults `datafusion.optimizer.prefer_hash_join` to `false`. For a SQL `NOT IN` predicate, DataFusion's physical planner can therefore replace the logical null-aware anti join with a `SortMergeJoinExec` before Ballista's scheduler rules run.
`SortMergeJoinExec` does not carry the `null_aware` flag. The scheduler cannot recover the required `NOT IN` NULL semantics afterward.
## Reproduction
Use four CSV partitions for each table:
- `t1(a)`: values `0..19`
- `t2(b)`: values `0..4`, with one key replaced by `NULL`
Run Ballista with its default join preference:
```sql
select a
from t1
where a not in (select b from t2)
order by a;
```
Expected result: **0 rows**, because the subquery contains `NULL`.
Observed result: **16 rows**. Single-process DataFusion returns the correct 0 rows.
Setting `datafusion.optimizer.prefer_hash_join = true` retains a null-aware `HashJoinExec`, so the scheduler can recognize and lower it correctly. Apache #2188 addresses that hash-join and adaptive-planning path, but cannot fix a null-aware flag that was already discarded.
## Expected behavior
Physical planning should retain null-aware anti-join semantics regardless of the global hash-join preference. Possible approaches include forcing a null-aware `HashJoinExec` for this logical shape or adding equivalent null-aware support to the selected physical join.
Discovered while validating #2188 from the end-to-end reproduction in @andygrove's review.
Contributor guide
Research direction
Start with the DataFusion physical planner path for SQL NOT IN and the Ballista scheduler rules that handle HashJoinExec and SortMergeJoinExec. Run the four-partition CSV reproduction with the default join preference, then trace where null-aware anti-join information is lost. Done means the query returns 0 rows regardless of the global hash-join preference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100