apache / apache/datafusion-comet
Publish nightly SNAPSHOT jars to repository.apache.org
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
Anyone who wants to try a fix or feature before the next Comet release has to build from source, including the Rust native library, which needs a Rust toolchain, protoc and a JDK. Downstream projects that want to test against Comet's main branch have the same problem.
Other ASF projects such as Spark and Iceberg publish nightly SNAPSHOT artifacts to https://repository.apache.org/content/repositories/snapshots/ from a scheduled GitHub Actions workflow, using the Nexus credentials that ASF Infra provisions as repository secrets.
The repository previously had a tag-triggered workflow that published Docker images to GHCR. It was removed in #4241 after failing for a long time: first within a minute of starting, later as `startup_failure` because it used third-party actions that were not pinned to a commit SHA, and it also built arm64 under QEMU with caching disabled.
### Describe the potential solution
Add a scheduled workflow that:
- builds `libcomet.so` for linux/amd64 and linux/aarch64 on native runners, inside an Ubuntu 20.04 container so the glibc baseline matches the release builder
- assembles the shaded `comet-spark` jars for Spark 3.4 (Scala 2.12), 3.5 (Scala 2.12), 4.0 and 4.1 (Scala 2.13) with both native libraries, the same way `dev/release/build-release-comet.sh` does
- deploys them with `./mvnw deploy` using the `NEXUS_USER` / `NEXUS_PW` repository secrets; the `org.apache:apache` parent pom already routes SNAPSHOT deploys to `apache.snapshots.https`
- skips nights where `main` has not changed
- supports a `dry_run` dispatch that builds and verifies the jars without touching Nexus, so the workflow can be exercised on a fork
The installation guide should document where the snapshots live, which artifacts exist, and that they are unreleased builds for testing only.
macOS libraries are out of scope: the x86_64 build needs the Apple SDK cross-compile path, which CI cannot use.
### Additional context
If the `NEXUS_USER` / `NEXUS_PW` secrets are not yet configured on this repository, an INFRA Jira ticket is needed before the first real publish succeeds.
Contributor guide
Research direction
Start by reading dev/release/build-release-comet.sh and the existing release workflow history referenced in #4241. Trace how ./mvnw deploy assembles the Spark-version jars and native libraries, then verify the scheduled and dry_run paths, unchanged-main skip, and snapshot publication requirements. Update the installation guide with the snapshot location, artifacts, and testing-only status.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, rust, scala
- Domain
- build-system, ci-cd, documentation, release
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100