apache / apache/datafusion-comet

Decide how the Delta Lake contrib jar is built, versioned, and published in releases

Open
#5,882 0 comments 0 reactions 0 assignees View on GitHub
documentation question requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Context

#5365 adds the Delta Lake scan contrib as a separate Maven module (`contrib/delta-spark`) that only enters the reactor under the `delta` profile and produces its own jar, `comet-contrib-delta-spark_`. The user guide and the module README tell users to put that jar on the classpath alongside `delta-spark`. This issue is for deciding how that jar reaches users, and more generally how table-format integrations are packaged and released, since Delta and Iceberg currently take different shapes.

## How the release tooling behaves today

- `dev/release/build-release-comet.sh` runs `./mvnw install` into a temporary local repository for six profile combinations (Spark 3.4 and 3.5 with Scala 2.12 and 2.13, then Spark 4.0 and 4.1). None of those invocations pass `-Pdelta`, so the contrib module is never built during a release.
- `dev/release/publish-to-maven.sh` does not run `mvn deploy`. It signs and uploads every `.jar` and `.pom` found under `org/apache/datafusion` in that local repository. Two things follow:
- `maven.deploy.skip` has no effect on what gets published. Whether the contrib jar ships depends only on whether `-Pdelta` is on the `install` lines.
- Anything `install` produces is published. #5365 adds an unconditional `test-jar` execution to `spark/pom.xml` (about 6.7 MB per profile) so that the contrib suites can reuse `CometTestBase`. As written it would ship six `-tests.jar` artifacts per release.
- The native library needs no release change. The release containers build with default Cargo features, and the `delta` feature (deletion-vector decoding plus the planner arm) is in the default set, so the released `libcomet` already carries everything the contrib jar needs.

## Questions to settle

1. **Should the contrib jar be published at all?** If yes, `-Pdelta` goes on the Spark 3.5, 4.0 and 4.1 `install` lines (the 3.4 pairing is `delta-core` 2.4 and does not compile). If no, the docs should say build-from-source is the only route and the `maven.deploy.skip=false` override in the contrib pom should go. An experimental feature that only exists if users build from source is unlikely to get real-workload testing.
2. **What Delta versions does each published jar support?** Each jar compiles against one Delta release per Spark line (3.3.2, 4.0.1, 4.3.1). Nothing verifies binary compatibility across Delta minors, and `pom.xml` already records that Delta 4.1.0 and 4.2.0 are incompatible with Spark 4.1.3. Publishing implies a support statement in the user guide.
3. **Should the spark test-jar be published?** Options: bind the `test-jar` execution inside the `delta` profile so it only exists when the contrib is built, or accept publishing it (there is partial precedent in the `-test-sources.jar`).
4. **Why is Delta packaged differently from Iceberg, and should it stay that way?** Iceberg support lives inside `comet-spark`: `iceberg-spark-runtime` is a test-scope dependency only and `IcebergReflection` addresses every Iceberg class by name, so there is no profile, no separate jar, and no dependency on a specific Iceberg release. The Delta contrib compiles against `delta-spark` internals (`DeltaParquetFileFormat`, `DeletionVectorDescriptor`, `DeltaColumnMapping`), which forces a `provided` dependency, one artifact per Spark-and-Delta pairing, and rules out Spark 3.4. That structural difference explains the module, the profile, the separate CI workflow and the opt-in config. The two consistent end states are (a) fold Delta into `comet-spark` behind reflection like Iceberg, or (b) keep the contrib shape and finish it (publish the jar, document the Delta pairing, gate the test-jar). Deciding this on purpose also settles part of the core-versus-contrib boundary question raised in the #5365 review and in #5411.

## Related

- #5365 (the contrib PR)
- #174 (Delta Lake support)
- #5411 (convergence of the two Delta read paths)

Contributor guide

Open the contributing guide

Research direction

Start by reading dev/release/build-release-comet.sh and dev/release/publish-to-maven.sh, then inspect contrib/delta-spark/pom.xml, spark/pom.xml, and the user guide. Trace which profiles and artifacts each release currently builds and publishes. Done means the project has a decided Delta packaging and support policy, with release configuration and documentation aligned to it.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, shell
Domain
build-system, devops, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.