Support Storage Partition Joins in Gluten
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Storage partition join([SPJ](https://issues.apache.org/jira/browse/SPARK-37375)) is a feature that relaxes more constraints on SMJ to allow bucket join. As currently Gluten will force using `ShuffledHashJoin` , some queries may lose perf which should benefit more from SMJ. To support SPJ in Gluten, we will:
1) Keep using SMJ if the join conditions and data partitions match SPJ and at most one side shuffle exists before SMJ. As investigated, we can't do this as the `StrategyOverrides`, where `forceUsingHashJoin` happens, is based on logical plan and we can't get information of pre shuffles. It's also not reasonable to change the `HashJoin` back to SMJ in physical plan as it may lack of necessary pre sort for SMJ. So to simplify this, we need user to understand the query and data partitions and just set `spark.gluten.sql.columnar.forceShuffledHashJoin` to `false`.
2) SMJ is supported in Velox but there may be some join conditions not supported fully. We will need fix when customers need.
Contributor guide
Assessment
This issue has not been assessed yet.