[CORE] Register native shuffles through backend API
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
Part of https://github.com/apache/incubator-gluten/issues/6920.
So far Gluten's native shuffle implementations, including VL, CH, VL + RSS are activated by specifying corresponding shuffle manager types in Spark configuration.
However if we want to mix different native shuffles in the same Spark session even in the same query plan, as Spark only allows one shuffle manager to be registered (see also a related [SPIP](https://issues.apache.org/jira/browse/SPARK-45792)) yet, we should enhance out current approach by providing a registry of native shuffles internally under an umbrella shuffle manager. Then query planner could choose an optimal one among all the candidates.
Contributor guide
Assessment
This issue has not been assessed yet.