CI: speed up the release GitHub Action — artifact reuse, cache isolation, and fan-out builds

Why is the release Action slow?

The -6 Release Process workflow (.github/workflows/cicd_6-release.yml) is documented at 25–35 min. Investigation found the cost is almost entirely repeated work, not tests — release runs no test suites at all.

Ranked sources of waste:

1. The project is compiled twice per release. build runs clean install -Dprod=true -DskipTests=true and uploads maven-repo; cicd_comp_release-phase.yml then downloads that same repo and runs the full reactor ./mvnw ... install again before publish.sh maven publishes from ~/.m2/repository. Javadoc generation is the only new output.
2. The release build deliberately opts out of the remote Maven build cache. cicd_comp_build-phase.yml marks the build-cache-* secrets required: false and notes "every release workflow" builds from scratch. PR / merge-queue / trunk all pass OVH_S3_BUILD_CACHE_BUCKET_*. (Remote build-output reuse for releases is deferred in this plan — see below.)
3. Unused artifacts are built, docker saved and uploaded on the release path: docker-image (~1 GB tar) and build-classes. Release has no test jobs to consume them; only maven-repo and docker-build-context are downloaded.
4. Two multi-arch (linux/amd64,linux/arm64) Docker builds per release through QEMU, each with its own setup-qemu/setup-buildx. Layer cache uses type=gha with no explicit scope, so buildx keys it to the git ref — a fresh release tag/branch is a cold cache every time.
5. verify-branch does a serial GitHub API loop (gh api commits/<sha>/pulls per commit, then gh pr view per PR).

Already-optimized (not in scope): dependency caches (Maven repo, wrapper, Node, pnpm, SDKMAN), shallow checkouts, buildx ignore-error on cache export.

Goal

Speed up primary and Java-variant releases by publishing the selected release run's Maven outputs without rebuilding, skipping unused test artifacts, and isolating Docker layer caches — while preserving existing shared-workflow defaults.

Phase 1 — remove duplicated work

1. Write regression tests before implementation

• Add .github/workflows/tests/release-build-reuse.test.sh, following the existing workflow-test convention (extract real shell blocks, stub Maven/Docker/publishing).
• Cover: shared defaults still produce test images + build classes; release settings keep maven-repo and docker-build-context but skip test-image build/save/upload and class uploads; primary + Java-variant builds install the exact Maven version later published; default/explicit/separator-prefixed suffixes resolve consistently across build, deployment, publication; artifact restoration uses the selected run ID with matching suffix, auth, and digest validation; reuse mode never invokes the second Maven install, and missing artifacts fail rather than publishing nothing; Docker cache scopes separate main/dev images and Java variants with matching import/export scopes.
• Get developer approval and confirm the new assertions fail (Red) before implementing.

2. Separate deployment-context generation from test-artifact production

• maven-job/action.yml: add generate-test-image (default 'true') and generate-build-classes (default 'true'). Keep generate-docker responsible for docker-build-context. Gate SDKMAN lookup, build-docker-image-from-archive, save-docker-image, upload-docker-image on generate-docker && generate-test-image; gate persist-build-classes on generate-artifacts && generate-build-classes. Preserve Maven-repo uploads and all dependency caching.
• Forward both flags as boolean inputs (default true) in cicd_comp_build-phase.yml.
• Do not disable Maven's Docker packaging or change docker.buildArchiveOnly — the deployment context must still exist.

3. Make the initial build produce publication-ready Java-variant coordinates

• Add a release-build input (default false) to the build workflow and Maven action.
• In release mode, apply the normalized Java suffix as -Dchangelist during the initial clean install; for primary releases without a Java override, empty changelist/suffix. (Required because the initial build currently only suffixes artifact names — the second install is what creates suffixed Maven coordinates today.)
• Expose the resolved suffix (no leading separator) via action → job → reusable-workflow outputs; use the existing java<major> fallback, preserve explicit suffixes, and make downstream callers consume this output. Fixes the existing java25 vs java-25 fallback mismatch.
• Pass changed shell inputs through env vars and validate before constructing Maven args.

4. Add an opt-in publication path that does not reinstall

• cicd_comp_release-phase.yml: add reuse-build-artifacts (boolean, default false).
• Wire Restore Maven Repository to run-id: ${{ inputs.artifact_run_id }}, github-token: ${{ secrets.GITHUB_TOKEN }}, the suffix-specific artifact name, digest-mismatch: error; in reuse mode restore into a clean Maven repo dir.
• Before publishing, validate the expected release-version directory and required core POM/JAR files; reject missing/empty/wrong-version outputs (don't rely on publish.sh, which can succeed when no matching version exists).
• Run Install Release Artifacts only when deployment is enabled and reuse mode is disabled (preserve legacy default). In reuse mode publish directly from the restored repo via the existing version-filtered publisher.
• Keep primary-release Javadoc generation/upload and other release ops intact; no substitute reactor build.

5. Opt both release workflows into the optimized path

• cicd_6-release.yml and cicd_7-release-java-variant.yml: Build → release-build: true, generate-test-image: false, generate-build-classes: false, keep generate-docker: true; Release → reuse-build-artifacts: true; pass the build's resolved suffix into deployment and release.
• Keep github.run_id as artifact source; preserve job dependencies.
• Do not enable previous-build discovery, pass remote Maven cache credentials, or enable the S3 build-output cache.

6. Isolate Docker caches without changing the backend

• deploy-docker/action.yml: add cache-scope (default buildkit, BuildKit's implicit default); use the same scope in cache-from: type=gha,scope=… and cache-to: type=gha,scope=…,mode=max,ignore-error=true.
• cicd_comp_deployment-phase.yml: add optional docker-cache-scope-prefix (default ''). When supplied, compute stable scopes from prefix + main/dev image identity + normalized Java variant (or primary) + SDKMAN Java version — excluding release version, SHA, run ID. When omitted, keep buildkit. Set prefix release in both release workflows.
• Preserve platforms, QEMU, pull: true, tags, registry destinations, and nonfatal cache export.

7. Validate and measure

• Run the regression script, ShellCheck on changed shell blocks, actionlint on changed workflows; validate composite-action input/output wiring.
• On an authorized nonproduction run, exercise primary + alternate Java: confirm the initial install emits the complete publication inventory (parent POMs, classifiers); inspect variant POM coordinates and bytecode; restore on a fresh runner and verify unchanged hashes; publisher dry-run to a test destination; generate primary Javadocs without the removed install; verify Docker contexts remain consumable and both images retain platforms/tags.
• Confirm release runs contain no second reactor install and no test-image/build-class artifacts. Compare step durations and artifact sizes vs. a recent comparable release; distinguish cold vs. warm Docker cache — do not promise a number without measurements.
• Do not run the full integration suite or dispatch a production release just to test.

8. Document causes, improvements, remaining limits

• docs/core/CICD_PIPELINE.md: release artifact contract, flags/defaults, validation procedure, measured results when available; explain the two Maven installs, unused test-image construction/transfer, and two multi-arch deployment builds — not test suites.
• docs/infrastructure/DOCKER_BUILD_PROCESS.md: scope scheme and limits.
• Document that GHA cache scope and ref visibility are separate: explicit scopes prevent overwrites but don't grant cross-branch/tag access.
• Record shared S3 Maven build-output reuse as deferred pending the #36947 flatten fix and provenance review.

Files to Modify

• .github/actions/core-cicd/maven-job/action.yml — independent artifact flags, release-mode changelist, resolved suffix output.
• .github/workflows/cicd_comp_build-phase.yml — forward new inputs and expose resolved suffix.
• .github/workflows/cicd_comp_release-phase.yml — selected-run restoration, validation, opt-in publication without reinstalling.
• .github/actions/core-cicd/deployment/deploy-docker/action.yml — explicit cache scope.
• .github/workflows/cicd_comp_deployment-phase.yml — optional release cache isolation.
• .github/workflows/cicd_6-release.yml — enable optimizations and forward resolved suffix.
• .github/workflows/cicd_7-release-java-variant.yml — equivalent variant configuration.
• docs/core/CICD_PIPELINE.md — release reuse contract and performance explanation.
• docs/infrastructure/DOCKER_BUILD_PROCESS.md — cache isolation and visibility limits.

New Files

• .github/workflows/tests/release-build-reuse.test.sh — regression coverage for flags, Maven coordinates, artifact selection, publication, and cache scopes.

Risks

• Variant correctness: removing the second install without moving its changelist override leaves variant Maven artifacts missing/misversioned.
• Javadocs: the clean release runner must still generate docs from restored dependencies without the removed install.
• Cache limits: explicit scopes prevent main/dev and variant collisions but do not guarantee warm caches across release refs.
• Older release tags: local composite actions load from checked-out source — historical tags may need a separate backport.
• Artifact completeness: the existing publisher can silently accept an empty selection; prepublish checks are essential.
• Remaining costs: QEMU builds, Javadocs, sequential GitHub API work in verify-branch, and publication remain. API parallelization, registry-cache migration, and shared compiled-output reuse are outside this phase.

Phase 2 — fan-out builds + a single fail-closed publish

Phase 1 removes the duplicated rebuild and the unused artifacts, but the critical path stays sequential. Most of what remains is independent and can run in parallel. Publication must not.

verify-branch ─┐
               ├─ build (Maven, once) ─┬─ javadocs ────┐
release-prepare┘                       ├─ cli-native ──┤
                                       ├─ docker/amd64 ┤ → publish (single,
                                       ├─ docker/arm64 ┤      fail-closed)
                                       └─ sbom ────────┘

Nodes

• build — the Maven reactor. Shared prerequisite for everything else; cannot be parallelized. Emits maven-repo + docker-build-context (Phase 1 keeps exactly these two and drops the test image and build-classes).
• javadocs — downloads maven-repo, runs javadoc:javadoc -pl :dotcms-core. Removes Javadoc generation from the publisher.
• cli-native — already a matrix today (cicd_comp_cli-native-build-phase.yml:80).
• docker/amd64, docker/arm64 — one platform per leg on native runners.
• sbom — needs the published image digest.
• publish — one job, one writer, fails closed.

Native arm64 — no QEMU

cicd_comp_deployment-phase.yml:255,299 builds linux/amd64,linux/arm64 through QEMU, twice (main + dev image). The repo is public, so ubuntu-24.04-arm runners are free. Build one platform per leg and merge with docker buildx imagetools create. Expected to be the largest single win in the Docker path — measure before claiming a figure. Note tags currently come from docker/metadata-action at push time ({{sha}}, identifier, custom, extra-tags), so the merge step must apply the same tag set to the manifest rather than re-deriving it.

The publisher must fail closed on an expected-asset manifest

Do not publish by enumerating whatever happens to be in artifacts/. The publisher starts from an explicit manifest (coordinates, platforms, file names) and fails if any expected asset is missing or empty.

This is not hypothetical — cicd_release-cli.yml already has the fan-in shape, and its glob publishes however many files matched:

if [[ ! -f "$JAR" ]]; then exit 1; fi
for zip in artifacts/cli-artifacts-*/*.zip; do publish.sh file ...; done

A leg that succeeded but produced nothing matches zero files and the job still exits 0. A leg that was skipped and a leg that produced an empty artifact are indistinguishable.

Hard constraint — exactly one Maven publisher

publish.sh maven is a read-modify-write against shared remote state: update_artifact_metadata() (publish.sh:187-256) lists versions from S3, rebuilds maven-metadata.xml, and writes it back. Two concurrent publishing legs both list, both write, last writer wins, and <versions>/<latest> can silently lose entries — Maven consumers then resolve wrong. Maven publication stays in one job. (Already true of the primary + java-variant pair.)

Hard constraint — all legs build the same thing

release-prepare creates a new commit, which is already why reuse-previous-build is broken. Every leg must be pinned to the release tag, not github.sha, or legs will disagree about what they are building. Each leg should assert the ref/sha it built.

Ordering inside the fan-in

Not one flat publish: Docker manifest → SBOM (pulls the released image) → promote-latest → labels/Slack.

Cost caveat

Every leg downloads maven-repo (GBs). With runners spread across six US regions, N downloads can cost more than the serial time saved. Measure — and consider passing only what a leg needs (e.g. core jar + pom) instead of the whole repository.

Precedent

cicd_release-cli.yml already runs build + build-cli in parallel and fans in to a single publishing job. It is the shape to follow, and its glob-publish is the specific weakness to fix on the way.

Prerequisite

Phase 1 steps 3 (build emits publication-ready coordinates) and 4 (validated, fail-closed artifact reuse) are prerequisites — without them this publishes on top of an unvalidated reuse path.

Phase 2 risks

• Moving publication across job boundaries is rollback-unsafe; legacy and LTS release workflows need auditing or explicit exclusion.
• Docker tag derivation happens at push time today; a manifest merge must reproduce the tag set exactly (sha, identifier, custom, extra-tags) or tags drift.
• Per-leg maven-repo download may dominate the win; measure.
• fail-fast/cancelled legs must be distinguishable from legs that produced nothing.
• verify-branch's serial gh api commits/<sha>/pulls loop is an independent win, not part of the fan-out.

Decisions already made

• Java variant suffix convention — java-25 is canonical. RELEASE_JAVA_VARIANT_SUFFIX is set to exactly java-25 in the repository variables (with RELEASE_JAVA_VARIANT_VERSION=25.0.2-ms), and that is what the deployment phase, deploy-docker and the published 25.x variant coordinates already use. maven-job and cicd_comp_release-phase.yml derived java25 instead, which is what made the build publish maven-repo-java25 while the deployment phase asked for maven-repo-java-25. Both were corrected to derive java-<major>. An explicitly supplied suffix is still passed through verbatim, so a caller passing java25 keeps it.
• The non-dashed java25 form is deprecated. It is no longer used; -java-25 is the only form. Any guidance still recommending java25 was corrected repo-wide (input descriptions in maven-job, cicd_comp_build_phase, cicd_comp_release-phase, cicd_6-release, cicd_3-trunk, cicd_4-nightly, cicd_comp_test-phase, and the tag examples in cicd_8-manual-deploy). An explicitly supplied suffix is still passed through verbatim and is never rewritten, so a legacy non-dashed value is not silently repointed.
• Coordinate/tag impact: this changes the derived artifact name only. The variant release passes RELEASE_JAVA_VARIANT_SUFFIX explicitly, so published Maven coordinates and Docker tags are unchanged. Non-release callers (cicd_8-manual-deploy, nightly) that pass java-version without artifact-suffix now resolve consistently.
• Java version source of truth: .sdkmanrc. The workflow java-version input is an override, never a second source. Nothing in the pipeline may hardcode a Java version: bumping .sdkmanrc must be sufficient to move the whole release path (SDKMAN base-image tag, Docker cache-scope key, and any derived artifact suffix) to the new version.
• Scope (b): remove the redundant Maven rebuild, eliminate unused test-image production, and isolate Docker caches — for both primary and Java-variant releases, preserving shared-workflow defaults.
• Compiled-output reuse (a): reuse Maven build outputs only from the selected release run. The shared S3 Maven build-output cache stays disabled for releases, deferred pending the #36947 flatten fix and provenance review.

Migration hazard

cicd_7-release-java-variant.yml cannot release a tag created before this change. Local reusable workflows and ./.github/actions/... resolve from the ref the dispatch runs against, but cicd_comp_build-phase.yml checks out ref: release_tag before uses: ./.github/actions/core-cicd/maven-job, so the action comes from the tag. An older tag therefore has no release-build support (no -Dchangelist, so no suffixed coordinates) while the new cicd_comp_release-phase.yml expects them — validation fails and blocks the release.

It fails closed and loudly rather than publishing something wrong, and cicd_6-release.yml is unaffected because it creates its tag from main during the run. Resolve by backporting or by not enabling reuse-build-artifacts for pre-change tags.

Review status

Steps 1-4 were reviewed by the reviewer agent (fable). Steps 5-6 and the suffix-convention fix are unreviewed — the reviewer agent is unavailable (OpenRouter credits exhausted and the Anthropic provider returning API key is invalid).

Related

• #36947 — CI: content-addressed build caching across nx, Maven and Docker (umbrella).
• Open defect: .claude/worktrees/cache-flatten-fix / issue-36947-cache-flatten-fix (4a4af92031 "force flatten on build-cache hits so installed POMs resolve").

---

Plan produced via scout + planner agents; not yet implemented.

Start with .github/workflows/tests/release-build-reuse.test.sh and the listed Maven, release, deployment, and release-workflow files; compare their existing inputs, outputs, artifact handling, and cache settings. Run the regression script, ShellCheck, and actionlint as wiring changes land. Done means release builds publish validated restored artifacts without a second install, unused artifacts are skipped, cache scopes are isolated, and the two documentation files describe the measured behavior and limits.