[CORE] Mixed backend(s)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
Note: It's not a goal to make CH backend + VL backend work together as of now ([reason](https://github.com/apache/incubator-gluten/pull/8143#issuecomment-2519635299)). If you got any requirement like this please let me know.
Currently only one single backend can be used at the same time in Gluten. This topic suggests a plug-able system to allow enabling multiple backends at compile time (at runtime in future, maybe). For example, eventually one could build Gluten with `-P backends-velox,backends-another` to enable both Velox backend and `another` backend that could possibly be added to Gluten code base in future. In the case when a query is being run, Gluten's query planner could choose the best path to implement the physical plan that may be a mixing of Velox sub-plans and `another` sub-plans.
The feature could drive Gluten to evolve from a hand-written middle layer to a development platform to offload Spark SQL with different data formats, acceleration libraries, or hardware architectures.
Prior to doing this, a bunch of preparation work should be done including essential code refactors.
Contributor guide
Assessment
This issue has not been assessed yet.