apache / apache/datafusion-ballista

Default join planning loses NOT IN null semantics in SortMergeJoinExec

Open
#2,193 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

## Bug

Ballista defaults `datafusion.optimizer.prefer_hash_join` to `false`. For a SQL `NOT IN` predicate, DataFusion's physical planner can therefore replace the logical null-aware anti join with a `SortMergeJoinExec` before Ballista's scheduler rules run.

`SortMergeJoinExec` does not carry the `null_aware` flag. The scheduler cannot recover the required `NOT IN` NULL semantics afterward.

## Reproduction

Use four CSV partitions for each table:

- `t1(a)`: values `0..19`
- `t2(b)`: values `0..4`, with one key replaced by `NULL`

Run Ballista with its default join preference:

```sql
select a
from t1
where a not in (select b from t2)
order by a;
```

Expected result: **0 rows**, because the subquery contains `NULL`.

Observed result: **16 rows**. Single-process DataFusion returns the correct 0 rows.

Setting `datafusion.optimizer.prefer_hash_join = true` retains a null-aware `HashJoinExec`, so the scheduler can recognize and lower it correctly. Apache #2188 addresses that hash-join and adaptive-planning path, but cannot fix a null-aware flag that was already discarded.

## Expected behavior

Physical planning should retain null-aware anti-join semantics regardless of the global hash-join preference. Possible approaches include forcing a null-aware `HashJoinExec` for this logical shape or adding equivalent null-aware support to the selected physical join.

Discovered while validating #2188 from the end-to-end reproduction in @andygrove's review.

Contributor guide

Open the contributing guide

Research direction

Start with the DataFusion physical planner path for SQL NOT IN and the Ballista scheduler rules that handle HashJoinExec and SortMergeJoinExec. Run the four-partition CSV reproduction with the default join preference, then trace where null-aware anti-join information is lost. Done means the query returns 0 rows regardless of the global hash-join preference.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.