apache / apache/datafusion-comet

Publish nightly SNAPSHOT jars to repository.apache.org

Open
#5,899 0 comments 0 reactions 0 assignees View on GitHub
enhancement requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

Anyone who wants to try a fix or feature before the next Comet release has to build from source, including the Rust native library, which needs a Rust toolchain, protoc and a JDK. Downstream projects that want to test against Comet's main branch have the same problem.

Other ASF projects such as Spark and Iceberg publish nightly SNAPSHOT artifacts to https://repository.apache.org/content/repositories/snapshots/ from a scheduled GitHub Actions workflow, using the Nexus credentials that ASF Infra provisions as repository secrets.

The repository previously had a tag-triggered workflow that published Docker images to GHCR. It was removed in #4241 after failing for a long time: first within a minute of starting, later as `startup_failure` because it used third-party actions that were not pinned to a commit SHA, and it also built arm64 under QEMU with caching disabled.

### Describe the potential solution

Add a scheduled workflow that:

- builds `libcomet.so` for linux/amd64 and linux/aarch64 on native runners, inside an Ubuntu 20.04 container so the glibc baseline matches the release builder
- assembles the shaded `comet-spark` jars for Spark 3.4 (Scala 2.12), 3.5 (Scala 2.12), 4.0 and 4.1 (Scala 2.13) with both native libraries, the same way `dev/release/build-release-comet.sh` does
- deploys them with `./mvnw deploy` using the `NEXUS_USER` / `NEXUS_PW` repository secrets; the `org.apache:apache` parent pom already routes SNAPSHOT deploys to `apache.snapshots.https`
- skips nights where `main` has not changed
- supports a `dry_run` dispatch that builds and verifies the jars without touching Nexus, so the workflow can be exercised on a fork

The installation guide should document where the snapshots live, which artifacts exist, and that they are unreleased builds for testing only.

macOS libraries are out of scope: the x86_64 build needs the Apple SDK cross-compile path, which CI cannot use.

### Additional context

If the `NEXUS_USER` / `NEXUS_PW` secrets are not yet configured on this repository, an INFRA Jira ticket is needed before the first real publish succeeds.

Contributor guide

Open the contributing guide

Research direction

Start by reading dev/release/build-release-comet.sh and the existing release workflow history referenced in #4241. Trace how ./mvnw deploy assembles the Spark-version jars and native libraries, then verify the scheduled and dry_run paths, unchanged-main skip, and snapshot publication requirements. Update the installation guide with the snapshot location, artifacts, and testing-only status.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust, scala
Domain
build-system, ci-cd, documentation, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.