apache / apache/auron

Add join-time residual condition support for native outer joins

Open
#2,195 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

**Describe**
Residual join conditions for outer joins cannot be handled correctly by simply performing a native key join and filtering the result afterward. For `LeftOuter`, `RightOuter`, and `FullOuter`, post-join filtering can break Spark's null-extension semantics. At the moment, Auron avoids this by rejecting native conversion when the join condition is non-empty.

**Describe the solution you'd like**
Add join-time residual condition support for native outer joins.

Unlike the inner-join issues, this should introduce an explicit optional residual-condition field in the native join protocol and evaluate that condition during join matching rather than as a post-join filter.

Expected scope:
- `LeftOuter`
- `RightOuter`
- `FullOuter`
- native sort-merge join
- native shuffled-hash join
- native broadcast hash join if feasible under the same protocol extension

The implementation should preserve Spark semantics for unmatched rows and null-extended outputs.

**Describe alternatives you've considered**
One alternative is to approximate support by using a post-join filter, but that is not semantically correct for outer joins.

Another alternative is to postpone outer joins entirely and only support inner joins with residual predicates. That is a valid incremental step, but it still leaves an important set of Spark join semantics unsupported.

**Additional context**
The current native join protocol only carries equality join keys and does not include a residual-condition field. This issue is expected to require coordinated changes across:
- planner/protobuf
- Spark-side native join plan construction
- native engine join execution

This issue should not include:
- broadcast nested loop joins
- pure non-equi joins without equality keys

Suggested validation:
- left/right/full outer joins with `equi keys + residual predicate`
- unmatched-row preservation
- null handling aligned with Spark
- fallback when the residual predicate cannot be converted natively

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.