[SUPPORT][SPARK][NATIVE] make hudi integrate into gluten/velox
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, The integration between spark and gluten/velox has made a good performance on parquet or lake format. And @vinothchandar also mentioned this in https://github.com/apache/hudi/pull/8679. So I think Hudi should take part in.
Here is a design I proposed in gluten before and some discussion: https://github.com/oap-project/gluten/issues/3378
Now, all the `scan` types that gluten has supported are file based, like `BatchScan` or `FileSourceScanExec`. Datasource provides the list of files during planning, then gluten pass them to the native library and the native reader (parquet/orc/...) loads them.
For hudi cow table without `hoodie.schema.on.read.enable`, it can return `HadoopFSRelation` (that's file based) when call `createRelation`. So maybe we can make this integration easily if the native reader can load the hudi files correctly.
But for other hudi tables, they return `HoodieBaseRelation` (with `BaseRelation`, `FileRelation`, `PrunedFilteredScan`) that will be transformed to `RowDataSourceScanExec` that's not supported in gluten. To solve this, maybe there are two ways:
1. to make gluten support it. IMO, it's not easy, and not a high-priority thing in gluten.
2. to make hudi be file-based scan. But mor table needs to merge data, and hudi use Spark DatasourceV1 interface that doesn't have the ability to merge data, I guess we have to migrate to DSV2 to use `BatchScan` which can use hudi-defined reader to load data. As well as, a native C++ Hudi Reader is required in velox. With these two, hudi mor tables can be queried in native env.
Gluten: https://github.com/oap-project/gluten
Velox: https://github.com/facebookincubator/velox
@vinothchandar @xushiyan
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.