CI: content-addressed build caching across nx, Maven and Docker

Proposal: content-addressed build caching across CI

Status: proposal, needs a storage decision
Context: follows the measurement work in #36942 / #36945

The principle

Nothing should be built twice from identical inputs — across runs, across
branches, across PRs.

Today every run rebuilds everything: the whole nx workspace, all 24 Maven
modules, and the Docker image, from scratch, on a fresh runner. The
Initial Artifact Build that does most of this is the serial prefix gating
all ~25 test jobs.

The fix is content-addressed caching: hash a unit of work's real inputs, and if
that hash has been seen, fetch the output instead of recomputing it. Three
layers, three tools.

Layer	Tool	Today
Frontend (nx run-many -t build)	nx remote cache	local only, discarded with the runner
Java modules (CLI, core, all 24)	Apache Maven Build Cache Extension	not used
Docker image	buildx registry cache	no layer cache at all

The three are independent and can land in parallel.

Why actions/cache cannot be the answer

GitHub Actions caches are branch-scoped. A run reads caches from its own
branch and the default/base branch — never a sibling branch. Two PRs building
identical code share nothing. That is a deliberate security boundary.

So actions/cache only supports save-on-trunk, restore-everywhere: a PR
benefits only from what main has already built.

Second problem: .nx/cache is 2.0 GB on a working machine. The Actions
cache is 10 GB per repo with LRU eviction, already shared with the Maven
repository, pnpm store and Node binary caches. A large, frequently-rewritten
entry would evict the others and could make builds slower.

A real remote cache is keyed purely on the content hash — no branch dimension,
no 10 GB ceiling.

Runner geography (measured, not assumed)

12 samples via PR #36948, cross-checked two ways (Azure IMDS .location and the
Azure Region: line GitHub prints in every job log header — they agreed 100%):

centralus       x3   Iowa
westus2         x3   Washington
eastus          x2   Virginia
northcentralus  x2   Illinois
westus          x1   California
eastus2         x1   Virginia

Runners are spread across six US Azure regions, coast to coast. No region
dominates. Consequences:

• A single-region bucket is the wrong shape — fast for roughly one job in
  six, slow for the rest (westus↔eastus is ~60–70ms RTT).
• This favours a managed multi-region cache, or a bucket behind a CDN. If
  self-hosting in one region anyway, pick a central compromise (us-east-2),
  not us-east-1.
• Small per-task entries (nx outputs, module jars) tolerate this well; large
  artifacts do not — see the Docker section.

Incidental finding: all runners are 4 vCPU / 15 GB, but the VM SKU varies by
generation (Standard_D4ads_v5 ×10, D4ds_v6 ×1, D4ds_v7 ×1). That is a
concrete cause of run-to-run wall-time noise, and a reason to judge any change
over several runs.

Layer 1 — frontend (nx)

Nx already hashes task inputs; it only lacks somewhere to keep results.

• Nx Cloud — free plan, multi-region edge (which the runner spread makes
  genuinely valuable), nothing to operate. Build outputs leave our
  infrastructure, so it needs a security review.
• Self-hosted — Nx ships S3/GCS/Azure adapters; self-hosted caching moved
  from paid Powerpack to free (activation key). We already have AWS
  credentials in CI (deploy-javadoc → s3://static.dotcms.com). Confirm the
  adapter's current packaging for Nx 23 before committing — the Nx 23 docs lead
  with Nx Cloud and with "build your own server".
• Custom server — stable OpenAPI contract (PUT/GET /v1/cache/{hash})
  plus NX_SELF_HOSTED_REMOTE_CACHE_SERVER and
  NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN. Only if the above are rejected.

Layer 2 — Java modules (Maven Build Cache Extension)

The Apache Maven Build Cache Extension
fingerprints each module's inputs by content digest, caches its outputs, and
skips the module on a hit. Remote cache works over plain HTTP PUT/GET/HEAD,
or anything Maven Wagon speaks (S3, SSH).

This is the answer to "we don't need to build the CLI every build." Rather
than hand-maintaining which modules a PR needs, the hash decides — and it covers
dotcms-cli, dotcms-api-data-model, dotcms-core and the other 21 modules at
once.

It also subsumes two other pieces of work:

• the Maven reactor trim still open on #36945, and
• the path-filter artifact reuse that #36081 attempted for the CLI.

Earlier I argued that reusing a prior WAR needs a provably complete predicate
for "did anything relevant change". Content hashing is that predicate,
computed rather than curated, which removes the class of bug where a hand-written
filter drifts and a PR silently tests a stale binary.

The real work: hidden inputs

Cache correctness depends on the hash covering every real input. Plugins that
read undeclared state produce hits that should have been misses. dotCMS has
several to handle explicitly:

• git metadata read during the build (git rev-parse for ShortRevision /
  scmBranch)
• openapi.yaml generated by swagger-maven-plugin at compile
• starter zip assembly
• the process-annotations immutables pass

Also enable project.build.outputTimestamp — timestamps embedded at package
time make outputs non-reproducible and silently destroy hit rate.

Layer 3 — Docker image

Currently there is no layer cache: every build runs docker build on a fresh
runner with an empty local cache. buildx registry caching
(--cache-to/--cache-from type=registry) fixes this and transfers only changed
layers.

Do not try to cache the image through the Maven or nx cache. The docker save
tar is ~1 GB, and with runners spread across six regions, pulling that from a
distant cache can cost more than rebuilding. Layer-level caching against a
registry is the right granularity.

What must NOT be cached (at least not first)

Test execution. The Maven extension caches package and later phases by
default and can cache test. Memoised test results mean a green run may mean
"we did not run", not "it passed". We have already measured real flakiness in
this suite, plus a hang that burned a 122m job timeout — masking that is worse
than the time it saves.

Cache compile/package. Run tests. Revisit only once hashing is demonstrably
correct.

Security

A cache that untrusted code can write is a remote code execution vector: a
poisoned entry is replayed as a build output on trunk.

• PR builds get read-only tokens. The nx OpenAPI contract returns 403 when
  a read-only token attempts a write; the Maven extension supports a read-only
  remote cache the same way.
• Only trusted branches write — trunk and release populate, PRs consume.
• Fork PRs never receive a write token.
• Hashes must include toolchain versions, so a Node or JDK bump cannot collide
  with an older entry.

This costs some hit rate — a PR cannot reuse another PR's novel work — but
keeps the case that matters: unchanged subsystems.

Order

1. nx remote cache. Biggest single win, best-understood tool, frontend only.
2. Maven build cache, package-and-later, read-only on PRs. Measure hit rate
   before trusting it; budget the real effort for hidden inputs above.
3. buildx registry cache for Docker. Independent of 1 and 2, and currently a
   complete gap.
4. Only then consider caching test phases.

Land #36946 first and re-measure — it stops building SDK projects the WAR never
contains, so it shrinks both the serial prefix and the 33m frontend job, and
today's baselines will not hold afterwards.

Start with the Initial Artifact Build and the nx run-many -t build entry point, then review the Maven Build Cache Extension and buildx --cache-to/--cache-from flow. Re-measure after #36946 before choosing storage. Done means compile/package outputs and Docker layers reuse content-addressed results securely, while tests still execute and hit rates are measured.