[SUPPORT] could hudi skip shuffle in SortMergeJoin, like what bucketby does in Spark?
Open
engine:spark
type:feature
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
As explained in https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4861715144695760/2994977456373837/5701837197372837/latest.html, the join of two bucketed tables could skip shuffle in a SortMergeJoin.
Is there anything similar in Hudi ? I think it could greatly improve join performance.
I tried bucketIndex, but the join between two tables with bucketIndex still needs shuffle.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.