[Bug] Core image build layer cache not applied

The Build core image workflow neither imports nor exports Docker layer cache, so every core build recompiles TheRock/ROCm and the full PyTorch stack from scratch. Observed core builds took 2h53m and 4h51m with zero cached steps. This is the same defect as #60, which covers the NV image workflow, plus one additional failure specific to docker/Dockerfile.ci.

Background

.github/workflows/build-core-image-reusable.yml builds docker/Dockerfile.ci on a remote Buildx builder (BUILDX_BUILDER) in two steps within one job:

1. Build core image — --target core, pushed as amdsiloai/pytorch-xdit-core:<tag>. Skipped when prebuilt_core_image_tag is supplied.
2. Build untuned image — the full Dockerfile (default final stage), pushed as amdsiloai/pytorch-xdit-staging:<tag>-temp.

Both steps take their cache source from the Determine images to build step:

if [ -n "$PREBUILT_CORE_TAG" ]; then
  echo "cache_source=${CORE_IMAGE_NAME}:${PREBUILT_CORE_TAG}" >> $GITHUB_OUTPUT
else
  echo "cache_source=${CORE_IMAGE_NAME}:${IMAGE_TAG}" >> $GITHUB_OUTPUT
fi

It is called by build-core-image.yml and by build-and-benchmark.yml.

Steps to reproduce

1. Run Build core image on any commit, leaving prebuilt_core_image_tag empty.
2. Run it again on a different commit.
3. In the Build xDiT images / Build images job log, search the Build core image step for importing cache manifest and CACHED.

Expected behavior

The second core build imports the previous build's layers from the registry and reports CACHED for every stage whose inputs are unchanged, rather than recompiling ROCm and PyTorch.

Actual behavior

The Build core image step imports nothing and caches nothing, and no cache is exported anywhere in the job.

Run	Commit	Core build step	CACHED steps	importing cache manifest	Longest single step
117	aee6940	14:12:42 → 17:05:19 (2h52m37s)	0	0	8490.2s (2h21m30s)
116	e8fd8f0	17:44:59 → 22:35:46 (4h50m47s)	0	0	12848.5s (3h34m)

Neither job contains a single exporting cache line.

The two jobs ran back to back on the build-only runner (run 117's job finished at 17:10, run 116's core build began at 17:44), which is the "consecutive builds on the same runner" case from #60.

Note one caveat on run 116: docker/Dockerfile.ci did change between e8fd8f0 and aee6940 (the TRITON_COMMIT / PYTORCH_*_COMMIT pins were bumped and the vision/audio remotes moved to the ROCm forks), so invalidation of build_torch_stack and everything after it is legitimate there. What is not legitimate is that stages which do not consume those ARGs — base_os, build_python, base, ffmpeg_builder — also produced zero hits. That is consistent with no cache being available at all rather than with normal invalidation.

Root cause

1. Cache is only imported from the tag being built. With prebuilt_core_image_tag empty, cache_source is ${CORE_IMAGE_NAME}:${IMAGE_TAG} — exactly the tag the step is about to push (-t "${CACHE_IMAGE}"). For a new commit that tag does not exist, so the imagetools inspect guard drops --cache-from entirely and the previous core image is never consulted. There is no commit-independent cache ref. Structurally identical to #60.

2. No cache is ever exported. Both steps rely on --build-arg BUILDKIT_INLINE_CACHE=1, which is only honoured by the classic dockerd builder and is a no-op for docker buildx build against a remote or container driver. There is no --cache-to. Nothing survives the builder's local cache.

3. Inline cache could not fix this even if the build-arg worked. This is the part that goes beyond #60. Dockerfile.ci is not a linear chain: the expensive work happens in discarded builder stages whose artifacts enter the image through COPY --from= and RUN --mount=type=bind,from=:

• build_python
• setup_therock
• build_rocm (TheRock / ROCm build — the 2h21m–3h34m step above)
• build_torch_stack (PyTorch, Triton, vision, audio, torchcodec)
• ffmpeg_builder

Those stages' layers are not part of the pushed image, so inline cache (mode=min semantics) can never represent them. Only --cache-to type=registry,mode=max can. For Dockerfile.cuda in #60 the stages are linear and mode=min would have been adequate; here it is structurally insufficient.

Why this has gone unnoticed

The Build untuned image step looks healthy, and that masks the problem. In run 117 it took 5m17s with 60 CACHED steps and one successful importing cache manifest from …/pytorch-xdit-core:aee6940.

That is misleading on two counts. Its cache_source resolves only because the preceding step just pushed that tag in the same job. And the CACHED hits include build_rocm, build_torch_stack, ffmpeg_builder and setup_therock — precisely the discarded builder stages that inline cache cannot carry — so those hits come from the builder's warm local cache, not from the registry import.

The consequence is that the prebuilt_core_image_tag path is the weakest one, despite existing to save time. When skip_core_build=true, the core build step is skipped, so there is no warm local cache from it, and the untuned build's only cache source is a registry image carrying no usable cache metadata for the builder stages. That path can rebuild the entire ROCm and PyTorch stack.

Ruled out

• cleanup-workspace pruning the builder. The job calls it with images: '' and containers: '', and its Docker cleanup step is gated on those being non-empty, so it never runs.
• Runner-local state as the intended mechanism. The step targets a remote builder and the existing comment says cache is imported "straight from the registry", so correctness should not depend on which runner or builder instance is reused.

Environment

• Image and tag or digest: amdsiloai/pytorch-xdit-core:<short-sha>, amdsiloai/pytorch-xdit-staging:<short-sha>-temp; base docker.io/library/ubuntu:24.04@sha256:69cecf4bbf72d2d44a9eef1b71fb98c7fb973d78af11399deccef19beb008ad9
• Git commit: aee6940 (run 117), e8fd8f0 (run 116)
• GPU model and architecture: N/A (build-time issue); image targets gfx942 and gfx950
• OS: GitHub Actions self-hosted runner, label build-only (DinD)
• ROCm or CUDA version: ROCm built from TheRock in build_rocm
• Driver version: N/A
• Relevant package versions: docker buildx on the pre-provisioned BUILDX_BUILDER; docker/login-action@v4; actions/checkout@v7

Regression

Yes, and it is the same commit that regressed the NV workflow in #60: 33d246f1 ("Revise core image build environment (#903)", 2026-06-10).

Before that commit this workflow exported cache, and notably already used mode=max for the core build:

# core build, pre-33d246f1
docker buildx build --builder ... --cache-from type=registry,ref=<cache_source> \
  --cache-to type=local,dest=/tmp/.buildx-cache,mode=max \
# other steps
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache,mode=min

33d246f1 switched to a raw docker buildx build against an external BUILDX_BUILDER and removed both the local cache-to and the local cache-from from both steps, leaving only the ineffective BUILDKIT_INLINE_CACHE=1 build-arg. The pre-existing mode=max is worth noting: it indicates the mode=max requirement described in root cause 3 was already understood here and was lost in the migration.

ffebfc51 ("Workflow hardening (#18)", 2026-08-19) then added the imagetools inspect guard around --cache-from. The guard is reasonable in itself, but it converts "import from a missing ref" into "no import at all", removing the last visible sign of misconfiguration.

For context, 03ef3ff9 ("Bug/Docker build cache issues (#747)", 2026-04-23) shows this workflow's cache configuration has needed repair before.

Reproducibility

Always, on every core build of a new commit. Confirmed in runs 116 and 117, both with zero cache imports, zero cache exports and zero cached steps in the core build.

Suggested fix

Same shape as the fix for #60: a commit-independent registry cache ref, imported from and exported to, replacing the inline-cache build-arg. mode=max is not optional here, per root cause 3.

CACHE_REF="${CORE_IMAGE_NAME}:buildcache"

# Keep the existence guard, but probe the stable cache ref as well as the
# per-commit source. --raw is needed so the check also works on a
# cache-only manifest.
CACHE_FROM_FLAGS=""
for REF in "${CACHE_REF}" "${CACHE_IMAGE}"; do
  if docker buildx imagetools inspect --raw "${REF}" >/dev/null 2>&1; then
    CACHE_FROM_FLAGS="${CACHE_FROM_FLAGS} --cache-from type=registry,ref=${REF}"
  fi
done

docker buildx build \
  ... \
  ${CACHE_FROM_FLAGS} \
  --cache-to "type=registry,ref=${CACHE_REF},mode=max,image-manifest=true,oci-mediatypes=true,ignore-error=true" \
  -t "${CACHE_IMAGE}" .

Points specific to this workflow:

• Apply it to both steps. The untuned build should import the shared ref in addition to cache_source, so that it still has cache when skip_core_build=true. Exporting from both steps to one shared ref is fine: the core build writes a subset and the untuned build then writes a superset.
• mode=max is the load-bearing option. Without it the build_rocm and build_torch_stack stages cannot be restored on a cold builder, which is where nearly all the time goes.
• Unlike #60 this workflow builds a single native platform, so no per-platform cache refs are needed. Keeping the ref in the core image's own repository lets the registry deduplicate its blobs against the pushed core image.
• ignore-error=true keeps a cache-export failure from failing the image push. image-manifest=true requires BuildKit 0.12 or newer.

Other

• Companion issue for the NV image workflow, with the same root cause and regression commit: #60
• Run 117: https://github.com/AMD-AGI/diffusion-models-inference/actions/runs/35106881329
• Run 116: https://github.com/AMD-AGI/diffusion-models-inference/actions/runs/35101377581
• Verification for a fix: run the workflow twice on different commits without a prebuilt core tag. The first run should log a cold miss plus a successful exporting cache to registry; the second should log importing cache manifest from <core image>:buildcache and report CACHED for build_rocm and build_torch_stack. A third run with prebuilt_core_image_tag set is worth checking separately, since that path has no warm local cache to fall back on.