apache / apache/hudi

[SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox

Open
#10,252 11 comments 11 reactions 1 assignee Claimed by @vinothchandar View on GitHub
area:performance area:sql type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, The integration between spark and gluten/velox has made a good performance on parquet or lake format. And @vinothchandar also mentioned this in https://github.com/apache/hudi/pull/8679. So I think Hudi should take part in.

Here is a design I proposed in gluten before and some discussion: https://github.com/oap-project/gluten/issues/3378

Now, all the `scan` types that gluten has supported are file based, like `BatchScan` or `FileSourceScanExec`. Datasource provides the list of files during planning, then gluten pass them to the native library and the native reader (parquet/orc/...) loads them.

For hudi cow table without `hoodie.schema.on.read.enable`, it can return `HadoopFSRelation` (that's file based) when call `createRelation`. So maybe we can make this integration easily if the native reader can load the hudi files correctly.

But for other hudi tables, they return `HoodieBaseRelation` (with `BaseRelation`, `FileRelation`, `PrunedFilteredScan`) that will be transformed to `RowDataSourceScanExec` that's not supported in gluten. To solve this, maybe there are two ways:
1. to make gluten support it. IMO, it's not easy, and not a high-priority thing in gluten.
2. to make hudi be file-based scan. But mor table needs to merge data, and hudi use Spark DatasourceV1 interface that doesn't have the ability to merge data, I guess we have to migrate to DSV2 to use `BatchScan` which can use hudi-defined reader to load data. As well as, a native C++ Hudi Reader is required in velox. With these two, hudi mor tables can be queried in native env.

Gluten: https://github.com/oap-project/gluten
Velox: https://github.com/facebookincubator/velox

@vinothchandar @xushiyan

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.