apache / apache/gluten

[CORE] Register native shuffles through backend API

Open
#8,060 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

Part of https://github.com/apache/incubator-gluten/issues/6920.

So far Gluten's native shuffle implementations, including VL, CH, VL + RSS are activated by specifying corresponding shuffle manager types in Spark configuration.

However if we want to mix different native shuffles in the same Spark session even in the same query plan, as Spark only allows one shuffle manager to be registered (see also a related [SPIP](https://issues.apache.org/jira/browse/SPARK-45792)) yet, we should enhance out current approach by providing a registry of native shuffles internally under an umbrella shuffle manager. Then query planner could choose an optimal one among all the candidates.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.