Add a Spark bundle that packages Apache DataFusion Comet
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem
Hudi has no bundle that ships a native execution engine. Anyone wanting [Apache DataFusion Comet](https://datafusion.apache.org/comet/) alongside Hudi has to assemble and version-match the jars themselves. Comet publishes one artifact per Spark minor version and, at 1.0.0, covers only Spark 3.4, 3.5, 4.0 and 4.1.
### Proposal
Publish `hudi-native-spark-bundle`, the contents of `hudi-spark-bundle` plus Comet, for the Spark versions Comet actually releases for.
Scope note: Comet does not read Hudi tables natively (it rejects Hudi's file format with `Unsupported file format HoodieFileGroupReaderBasedFileFormat`), so the Parquet decode stays on Spark. What the bundle enables is native execution of the rest of the plan: joins, shuffles, sorts and aggregates over Hudi tables. Making the Hudi scan itself native is separate work.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.