apache / apache/gluten

[VL] Unified design for data lake read support in Gluten + Velox

Open
#3,378 33 comments 7 reactions 0 assignees View on GitHub
DATA_LAKE enhancement on track
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Currently there are 2 PRs opened in Gluten to support [Iceberg COW table read](https://github.com/oap-project/gluten/pull/3043) and [Delta Lake read](https://github.com/oap-project/gluten/pull/2902). There is also one hot discussion in Velox about [Iceberg read](https://github.com/facebookincubator/velox/issues/5977) support. By consolidating the ideas and based on Gluten's position, we would like to share a draft unified design for data lake read support in Gluten.

As addressed in this project's home page, one of Gluten key function is to transform Spark’s whole stage physical plan to Substrait plan and send to native. It applies to data lake read support, thus:

1. We'd better avoid hacking of original Spark physical plan node. Gluten core has plan transformer to generate correct plan info into substrait format and then pass it to Velox for read and computation. So no matter what kind of the hack is, it should can be done in the transformer layer, such as column mapping. IMO, we should try best pass original info in spark plan to Velox as a bridge and do correct consumption at Velox side, unless it's not doable or velox can't support. By the way, one issue for feature like column mapping is, it's a common feature for kinds of file format reading, velox can handle this at its datasource level and the community has plan to do so.

3. Clear transformer hierarchy is need for different data lake backends. In the [Iceberg COW table read](https://github.com/oap-project/gluten/pull/3043) PR, a new branch is added to do specific process for Iceberg and leverage an utility class put in a dedicated folder, and in future, I believe more branches will be needed to support other cases, like MOR. So introducing a new transformer inherited from `BatchScanExecTransformer` would be a better way. The possible hierarchy should be like following:

```
IcebergDataSoure? DeltaLakeDatasoure? ?
\ | /
\ | /
\ | /
velox
|
substrait
/ | \
/ | \
SparkBatchQueryScanExecTransformer DeltaLakeScanExecTransformer \
| |
BatchScanExecTransformer FileSourceScanExecTransformer HiveTableScanExecTransformer
\ | /
BasicScanExecTransformer
```
@YannByron @felipepessoto @liujiayi771 @ulysses-you, please give comments on above suggestions. Thanks.

Contributor guide

Open the contributing guide

Research direction

Compare the Iceberg COW and Delta Lake PRs with the Velox Iceberg discussion, then review the proposed transformer hierarchy around BatchScanExecTransformer and the named scan transformers. Done means the project has an agreed unified design for data lake reads, including transformer boundaries and the Gluten-to-Velox/Substrait flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.