apache / apache/gluten

[CORE] Mixed backend(s)

Open
#6,920 0 comments 6 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

Note: It's not a goal to make CH backend + VL backend work together as of now ([reason](https://github.com/apache/incubator-gluten/pull/8143#issuecomment-2519635299)). If you got any requirement like this please let me know.

Currently only one single backend can be used at the same time in Gluten. This topic suggests a plug-able system to allow enabling multiple backends at compile time (at runtime in future, maybe). For example, eventually one could build Gluten with `-P backends-velox,backends-another` to enable both Velox backend and `another` backend that could possibly be added to Gluten code base in future. In the case when a query is being run, Gluten's query planner could choose the best path to implement the physical plan that may be a mixing of Velox sub-plans and `another` sub-plans.

The feature could drive Gluten to evolve from a hand-written middle layer to a development platform to offload Spark SQL with different data formats, acceleration libraries, or hardware architectures.

Prior to doing this, a bunch of preparation work should be done including essential code refactors.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.