apache / apache/iggy

Publish downloadable binary artifacts and connector plugins for stable releases (GitHub Releases)

Open
#4,038 1 comment 0 reactions 0 assignees View on GitHub
CI/CD connectors enhancement
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

## Motivation

Apache Iggy currently publishes compiled binaries (`iggy-server`, `iggy`, `iggy-bench`, `iggy-connectors`) and connector plugin `.so` shared libraries **only** to the rolling `:edge` GitHub pre-release on master commits (`.github/workflows/edge-release.yml`).

When official/versioned releases are published via `.github/workflows/publish.yml`, artifacts are distributed to package registries:
- Rust crates -> crates.io
- Container images -> Docker Hub (`apache/iggy`, `apache/iggy-connect`, etc.)
- Python SDK -> PyPI
- Node SDK -> npm
- Java SDK -> Maven Central / ASF Nexus
- C# SDK -> NuGet
- Go SDK -> Git tag

However, **GitHub Releases for versioned tags (e.g. `server-0.8.0`, `cli-0.8.0`, etc.) do not include downloadable tarballs / binary assets**. Users who want standalone binaries without Docker, `cargo install`, or building from source cannot currently download pre-compiled tarballs from GitHub Releases. (AI Disclosure: GitHub copilot helped in the investigation and drafting the issue)

Providing downloadable binary tarballs and connector plugins on GitHub Releases significantly improves developer adoption, simplifies CI integration in downstream projects, and provides a quick onboarding path for users evaluating Iggy without Docker.

---

## Current Architecture & Existing Components

The repository already has all the required building blocks in place:

1. **Reusable Build Workflow (`.github/workflows/_build_rust_artifacts.yml`)**:
- Compiles release binaries across cross-platform matrix:
- `x86_64-unknown-linux-gnu` (glibc)
- `x86_64-unknown-linux-musl` (musl static)
- `aarch64-unknown-linux-gnu` (glibc)
- `aarch64-unknown-linux-musl` (musl static)
- Compiles connector plugins (`.so` shared libraries) for:
- `x86_64-unknown-linux-gnu`
- `aarch64-unknown-linux-gnu`
- Generates tarballs:
- `iggy-${target}-${version}.tar.gz` (contains `iggy-server`, `iggy`, `iggy-bench`, `iggy-connectors`)
- `iggy-connectors-${target}-${version}.tar.gz` (contains plugin `.so` files)
- Packages and uploads a combined artifact `rust-artifacts-all`.

2. **Edge Release Workflow (`.github/workflows/edge-release.yml`)**:
- Calls `_build_rust_artifacts.yml` with `version: edge`.
- Downloads `rust-artifacts-all`.
- Uses `softprops/action-gh-release@v3.0.2` to upload `artifacts/*.tar.gz` to the rolling `edge` pre-release.

3. **Release Publisher (`.github/workflows/publish.yml`)**:
- Central entry point for manual (`workflow_dispatch`) and automated post-merge releases.
- Pushes Docker manifests, publishes packages to crates.io / PyPI / npm / NuGet / Maven, and creates corresponding Git tags using `.github/actions/utils/create-git-tag`.
- **Missing step**: It does not build or upload the compiled Rust binary/plugin tarballs as GitHub Release assets.

---

## What is Needed to Enable Release Binary Artifacts

To attach downloadable tarballs to tagged releases, the following additions are needed:

### 1. Update `.github/workflows/publish.yml`
- When `publish_dockerhub` includes `rust-server` or when server/CLI releases are published:
- Invoke `_build_rust_artifacts.yml` passing `version: ${{ steps.ver.outputs.version }}` and `commit: ${{ needs.validate.outputs.commit }}`.
- Add a release step (using `gh release create` / `gh release upload` or `softprops/action-gh-release`) to create/update the GitHub Release for the server/CLI tag (e.g. `server-vX.Y.Z` or `vX.Y.Z`) and attach the generated tarballs (`iggy-*.tar.gz`, `iggy-connectors-*.tar.gz`).
- Generate and attach SHA256 checksums (`SHA256SUMS.txt`) for all tarballs to adhere to ASF release integrity recommendations.

### 2. Optional: Add Windows and macOS Binary Targets (when support matures)
- Once Windows native compilation blockers (#) and macOS targets are addressed:
- Add `x86_64-pc-windows-msvc` and `x86_64-apple-darwin` / `aarch64-apple-darwin` matrix rows to `_build_rust_artifacts.yml`.
- Package `.zip` for Windows and `.tar.gz` for macOS/Linux.

### 3. Checksum & Provenance
- Include a checksum generation step:
```bash
sha256sum *.tar.gz > SHA256SUMS.txt
```
- Optionally enable GitHub Artifact Attestations (`actions/attest-build-provenance`) for supply chain security.

---

## Suggested Acceptance Criteria

- [ ] Publishing a release via `publish.yml` creates a GitHub Release (or updates the tag's release) containing downloadable `.tar.gz` tarballs for Linux x86_64 and aarch64 (glibc and musl).
- [ ] Each release includes `SHA256SUMS.txt` for integrity verification.
- [ ] Tarballs contain all primary binaries (`iggy-server`, `iggy`, `iggy-bench`, `iggy-connectors`) and connector plugins.
- [ ] Download links are documented or visible on the GitHub Releases page for easy end-user consumption.

### Affected area / component

_No response_

### Proposed solution

_No response_

### Alternatives considered

_No response_

### Contribution

- [ ] I'm willing to submit a pull request to implement this feature

### Good first issue

- [ ] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Research direction

Start by comparing .github/workflows/publish.yml with .github/workflows/edge-release.yml, then read .github/workflows/_build_rust_artifacts.yml and the create-git-tag action. Trace how publish.yml determines the version, commit, and release tag before checking the artifact download and upload flow. Done means versioned GitHub Releases contain the Linux tarballs, connector plugins, and SHA256SUMS.txt described in the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
build-system, ci-cd, release
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.