apache / apache/gluten

Support Storage Partition Joins in Gluten

Open
#4,571 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Storage partition join([SPJ](https://issues.apache.org/jira/browse/SPARK-37375)) is a feature that relaxes more constraints on SMJ to allow bucket join. As currently Gluten will force using `ShuffledHashJoin` , some queries may lose perf which should benefit more from SMJ. To support SPJ in Gluten, we will:
1) Keep using SMJ if the join conditions and data partitions match SPJ and at most one side shuffle exists before SMJ. As investigated, we can't do this as the `StrategyOverrides`, where `forceUsingHashJoin` happens, is based on logical plan and we can't get information of pre shuffles. It's also not reasonable to change the `HashJoin` back to SMJ in physical plan as it may lack of necessary pre sort for SMJ. So to simplify this, we need user to understand the query and data partitions and just set `spark.gluten.sql.columnar.forceShuffledHashJoin` to `false`.
2) SMJ is supported in Velox but there may be some join conditions not supported fully. We will need fix when customers need.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.