Add join-time residual condition support for native outer joins
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
**Describe**
Residual join conditions for outer joins cannot be handled correctly by simply performing a native key join and filtering the result afterward. For `LeftOuter`, `RightOuter`, and `FullOuter`, post-join filtering can break Spark's null-extension semantics. At the moment, Auron avoids this by rejecting native conversion when the join condition is non-empty.
**Describe the solution you'd like**
Add join-time residual condition support for native outer joins.
Unlike the inner-join issues, this should introduce an explicit optional residual-condition field in the native join protocol and evaluate that condition during join matching rather than as a post-join filter.
Expected scope:
- `LeftOuter`
- `RightOuter`
- `FullOuter`
- native sort-merge join
- native shuffled-hash join
- native broadcast hash join if feasible under the same protocol extension
The implementation should preserve Spark semantics for unmatched rows and null-extended outputs.
**Describe alternatives you've considered**
One alternative is to approximate support by using a post-join filter, but that is not semantically correct for outer joins.
Another alternative is to postpone outer joins entirely and only support inner joins with residual predicates. That is a valid incremental step, but it still leaves an important set of Spark join semantics unsupported.
**Additional context**
The current native join protocol only carries equality join keys and does not include a residual-condition field. This issue is expected to require coordinated changes across:
- planner/protobuf
- Spark-side native join plan construction
- native engine join execution
This issue should not include:
- broadcast nested loop joins
- pure non-equi joins without equality keys
Suggested validation:
- left/right/full outer joins with `equi keys + residual predicate`
- unmatched-row preservation
- null handling aligned with Spark
- fallback when the residual predicate cannot be converted natively
Contributor guide
Assessment
This issue has not been assessed yet.