apache / apache/gluten

[CORE] Doc: Add developer guide for adding new backend / component for Gluten

Open
#9,958 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

Beginning with https://github.com/apache/incubator-gluten/issues/7031, after months of iteration on the new `gluten-core` module, we are basically have an ready-to-use Spark accelerator SPI extracted out from the old Gluten code base.

Now (after https://github.com/apache/incubator-gluten/pull/9900) `gluten-core` is a thin Spark SPI layer that is not directly relevant to specific accelerator technologies and their programming languages, e.g., C++ / Substrait / Velox / ClickHouse, etc.

We'll add some developer-oriented documents showing how to add a new backend in Gluten, E.g., a backend (component) for accelerating a specific lake-format processing, or a backend with brand-new native accessing layer or programming language, and how to make them cooperate smoothly with the existing Velox / ClickHouse backend in Gluten.

Contributor guide

Open the contributing guide

Research direction

Start with the gluten-core module, the changes in pull request 9900, and the earlier discussion in issue 7031. The guide is done when it explains how to add a new backend or component, including a native access layer or programming language, and how it cooperates with the existing Velox and ClickHouse backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, cpp, scala
Domain
backend, documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.