apache / apache/datafusion-comet

[EPIC] Add support for all join operators

Open
#12 7 comments 4 reactions 1 assignee Claimed by @coderfender View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

This is an umbrella ticket for adding Join support to Comet. In Spark, there are basically three types of Join operators: BroadcastJoin, HashJoin, SortMergeJoin. In DataFusion, two Join operators are supported: HashJoin, SortMergeJoin (experimental).

We are going to delegate Spark Join operators to correspond DataFusion Join operators. We will also go to improve DataFusion Join operators if needed.

### SortMergeJoin

- [x] Relax join keys constraint from Column to any physical expression for physical join operators https://github.com/apache/arrow-datafusion/pull/8991
- [x] Support join filter for SortMergeJoin https://github.com/apache/arrow-datafusion/pull/9080
- [x] Add spilling in SortMergeJoin https://github.com/apache/arrow-datafusion/issues/9359
- [x] Support SortMergeJoin operator - #177
...

### HashJoin

- [x] Support HashJoin operator - #193

### BroadcastJoin

- [x] Add CometBroadcastExchangeExec support broadcasting the result of Comet native operator - https://github.com/apache/arrow-datafusion-comet/issues/81
- [x] Support BroadcastHashJoinExec #202
- [x] Enable Comet broadcast by default #212

### Other Join operators (Cross Join)

- [ ] Support CartesianProductExec - #199
- [x] Support BroadcastNestedLoopJoinExec - #198

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.