[CORE] Doc: Add developer guide for adding new backend / component for Gluten
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
Beginning with https://github.com/apache/incubator-gluten/issues/7031, after months of iteration on the new `gluten-core` module, we are basically have an ready-to-use Spark accelerator SPI extracted out from the old Gluten code base.
Now (after https://github.com/apache/incubator-gluten/pull/9900) `gluten-core` is a thin Spark SPI layer that is not directly relevant to specific accelerator technologies and their programming languages, e.g., C++ / Substrait / Velox / ClickHouse, etc.
We'll add some developer-oriented documents showing how to add a new backend in Gluten, E.g., a backend (component) for accelerating a specific lake-format processing, or a backend with brand-new native accessing layer or programming language, and how to make them cooperate smoothly with the existing Velox / ClickHouse backend in Gluten.
Contributor guide
Research direction
Start with the gluten-core module, the changes in pull request 9900, and the earlier discussion in issue 7031. The guide is done when it explains how to add a new backend or component, including a native access layer or programming language, and how it cooperates with the existing Velox and ClickHouse backends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, cpp, scala
- Domain
- backend, documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100