apache / apache/datafusion-comet
[EPIC] Add support for all join operators
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
This is an umbrella ticket for adding Join support to Comet. In Spark, there are basically three types of Join operators: BroadcastJoin, HashJoin, SortMergeJoin. In DataFusion, two Join operators are supported: HashJoin, SortMergeJoin (experimental).
We are going to delegate Spark Join operators to correspond DataFusion Join operators. We will also go to improve DataFusion Join operators if needed.
### SortMergeJoin
- [x] Relax join keys constraint from Column to any physical expression for physical join operators https://github.com/apache/arrow-datafusion/pull/8991
- [x] Support join filter for SortMergeJoin https://github.com/apache/arrow-datafusion/pull/9080
- [x] Add spilling in SortMergeJoin https://github.com/apache/arrow-datafusion/issues/9359
- [x] Support SortMergeJoin operator - #177
...
### HashJoin
- [x] Support HashJoin operator - #193
### BroadcastJoin
- [x] Add CometBroadcastExchangeExec support broadcasting the result of Comet native operator - https://github.com/apache/arrow-datafusion-comet/issues/81
- [x] Support BroadcastHashJoinExec #202
- [x] Enable Comet broadcast by default #212
### Other Join operators (Cross Join)
- [ ] Support CartesianProductExec - #199
- [x] Support BroadcastNestedLoopJoinExec - #198
Contributor guide
Assessment
This issue has not been assessed yet.