[None][infra] Add Docker BuildKit cache (--cache-to/--cache-from) to reduce Docker image build times

Summary

The Docker image build pipeline does not use BuildKit's external cache storage (--cache-to / --cache-from). Every build—on every Kubernetes pod, every PR, every post-merge—starts from scratch with no Docker layer cache. Adding multi-layer registry-based caching could significantly reduce image build times across all CI/CD pipelines.

Current State

How images are built today

The build system uses docker buildx build via docker/Makefile with these default options:

DOCKER_BUILD_OPTS ?= --pull --load

No --cache-to or --cache-from flags are passed anywhere:

• docker/Makefile — the %_build target calls docker buildx build without any cache export/import directives
• jenkins/BuildDockerImage.groovy — invokes make -C docker <target>_<action> without cache flags
• jenkins/Build.groovy — builds wheels in-container, no Docker layer caching
• .github/workflows/ — no Docker image builds in GitHub Actions

What is cached today

Cache Type	Mechanism	Scope
C++ compilation	ccache (4.9.1) on PVC at /mnt/sw-tensorrt-pvc/scratch.trt_ccache/llm_ccache	Persistent across builds, 500GB limit
pip packages	BuildKit --mount=type=cache,target=/root/.cache/pip	Per-builder ephemeral (lost when pod dies)
Docker layers	None	Rebuilt from scratch on each new pod

Images built per pipeline

BuildDockerImage.groovy builds 9 images in parallel per pipeline run:

1. Build Internal release (x86_64 trtllm) — release stage
2. Build Internal release (SBSA trtllm) — release stage
3. Build CI Image (x86_64 tritondevel) — tritondevel stage
4. Build CI Image (SBSA tritondevel) — tritondevel stage
5. Build CI Image (RockyLinux8 Python310) — tritondevel stage
6. Build CI Image (RockyLinux8 Python312) — tritondevel stage
7. Build CI Image (SBSA Ubuntu24.04 Python312) — tritondevel stage
8. Build NGC devel And release (x86_64) — devel + release stages
9. Build NGC devel And release (SBSA) — devel + release stages

Each runs on a fresh Kubernetes pod with Docker-in-Docker (docker:dind), meaning zero Docker layer cache is available at build start.

Dockerfile.multi stage structure

base (nvcr.io/nvidia/pytorch:26.02-py3)
  └─ devel (TRT, CUDA toolkit, ccache, MPI4py, UCX, NIXL, etcd)
       ├─ tritondevel (+ Triton server + Mooncake + CI tools)
       ├─ wheel (source COPY + build_wheel.py)
       │    ├─ release (pip install wheel + benchmarks + examples)
       │    │    └─ tritonrelease (+ Triton backend)
       │    └─ tritonbuild (Triton backend compilation)
       └─ (ngc-devel / ngc-release variants)

The devel stage is the most expensive — it installs TensorRT, CUDA toolkit, CMake, ccache, PyTorch, OpenCV, MPI4py, UCX, NIXL, etcd, and Mooncake. This stage rarely changes between commits.

Proposal

Add --cache-to and --cache-from to Docker buildx builds

Use BuildKit's registry-based cache backend to export and import layer caches. This allows builds on ephemeral Kubernetes pods to benefit from cached layers from previous builds.

Suggested implementation

1. Add cache variables to docker/Makefile:

DOCKER_CACHE_REPO  ?=
DOCKER_CACHE_OPTS  ?= $(if $(DOCKER_CACHE_REPO),\
  --cache-to type=registry,ref=$(DOCKER_CACHE_REPO):cache-$(IMAGE_TAG),mode=max \
  --cache-from type=registry,ref=$(DOCKER_CACHE_REPO):cache-$(IMAGE_TAG) \
  --cache-from type=registry,ref=$(DOCKER_CACHE_REPO):cache-main \
  --cache-from type=registry,ref=$(DOCKER_CACHE_REPO):cache-latest)

Then add $(DOCKER_CACHE_OPTS) to the %_build target's docker buildx build command.

2. Pass cache parameters from Jenkins (BuildDockerImage.groovy):

args += " DOCKER_CACHE_REPO=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm-cache"

3. Multi-layer cache strategy (3 levels):

Cache Level	Tag Pattern	Purpose
Branch-specific	cache-<branch_tag>	Fastest match for repeat builds on the same branch
Main branch	cache-main	Fallback for new branches — most layers will match
Latest stable	cache-latest	Updated after each successful post-merge build

Multiple --cache-from sources are tried in order; BuildKit uses the best match automatically.

4. Per-stage cache keys (optional optimization):

For the multi-stage Dockerfile, separate cache tags per stage maximize reuse:

--cache-from type=registry,ref=$(DOCKER_CACHE_REPO):cache-devel-main \
--cache-from type=registry,ref=$(DOCKER_CACHE_REPO):cache-tritondevel-main \
--cache-from type=registry,ref=$(DOCKER_CACHE_REPO):cache-wheel-$(BRANCH_TAG) \
--cache-to type=registry,ref=$(DOCKER_CACHE_REPO):cache-$(STAGE)-$(BRANCH_TAG),mode=max

This is useful because the devel stage changes infrequently while the wheel stage changes on every commit.

Expected impact

Stage	Typical Duration	With Cache
devel (install TRT, CUDA, UCX, NIXL, etc.)	Heavy	Fully cached when base image + install scripts unchanged
tritondevel (Triton server copies + installs)	Moderate	Cached when Triton version unchanged
wheel (C++ compilation + wheel build)	Heavy	Partially cached (source COPY invalidates, but ccache already helps here)
release (pip install wheel)	Light	Cached when devel unchanged

The devel stage is the highest-value target — it's expensive, shared by most images, and changes rarely.

Implementation considerations

• mode=max exports all layers (not just final), maximizing cache hits for multi-stage builds
• Registry auth — the cache registry requires the same URM credentials already used in the pipeline
• Cache size — registry-based cache is stored as image layers; old tags can be pruned on a schedule
• Backwards compatibility — when DOCKER_CACHE_REPO is empty (default), no cache flags are added, so local builds are unaffected
• Docker-in-Docker compatibility — type=registry cache works with DinD since it uses the Docker daemon's registry access (no local filesystem required)

Files to modify

1. docker/Makefile — add DOCKER_CACHE_REPO, DOCKER_CACHE_OPTS variables and inject into %_build
2. jenkins/BuildDockerImage.groovy — pass DOCKER_CACHE_REPO and branch-specific cache tags via args
3. jenkins/Build.groovy — if any Docker builds are added here in future, same pattern applies

References

• Docker BuildKit cache backends documentation
• BuildKit registry cache
• Multi-stage build caching best practices

Start with the %_build target in docker/Makefile and the make invocation in jenkins/BuildDockerImage.groovy; trace how image and branch tags are passed through the nine parallel builds. Verify registry cache import/export with ephemeral Docker-in-Docker builders, while confirming that an empty DOCKER_CACHE_REPO preserves local behavior and that the pipeline completes successfully.