apache / apache/hudi

Add a Spark bundle that packages Apache DataFusion Comet

Open
#19,673 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:build area:query-engine priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Problem

Hudi has no bundle that ships a native execution engine. Anyone wanting [Apache DataFusion Comet](https://datafusion.apache.org/comet/) alongside Hudi has to assemble and version-match the jars themselves. Comet publishes one artifact per Spark minor version and, at 1.0.0, covers only Spark 3.4, 3.5, 4.0 and 4.1.

### Proposal

Publish `hudi-native-spark-bundle`, the contents of `hudi-spark-bundle` plus Comet, for the Spark versions Comet actually releases for.

Scope note: Comet does not read Hudi tables natively (it rejects Hudi's file format with `Unsupported file format HoodieFileGroupReaderBasedFileFormat`), so the Parquet decode stays on Spark. What the bundle enables is native execution of the rest of the plan: joins, shuffles, sorts and aggregates over Hudi tables. Making the Hudi scan itself native is separate work.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.