Pushing cache to "registry" cache with multi-node builder only uploads cache from one node

Originally reported at https://github.com/moby/buildkit/issues/2758

It's been confirmed by @tonistiigi that this is a problem with buildx multi-node builder.

When you are building a multi-platform image with multiple builders (to avoid emulation) and use --cache-to type=registry, the resulting registry cache only contains cache for the platform that that was build last.

I tried to utilize buildkit to build Apache Airflow (https://github.com/apache/airflow) multi-platform images. I am using latest buildkit and latest docker:.

Hosts used for the multi-platform builds

I have two builder hosts:

1. AMD builder (Linux Mint 20.3) with buildx plugin installed
   github.com/docker/buildx v0.7.1 05846896d149da05f3d6fd1e7770da187b52a247 - docker builder created there

2. ARM Builder (Mac Pro M1 late 2021) with DockerDesktop 4.6.0 (with buildx pre-install installed) - with new Virtualization framework enabled.

Builder configuration

I configured my buildx builds to use both builders. I connected the MacOS builder to the Linux Host via forwarded docker socket and I am running all my multi-platform builds from the Linux Host.

This is the builders I see with docker buildx ls:

airflow_cache       docker-container                     
  airflow_cache0    unix:///var/run/docker.sock running  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64
  airflow_cache1    tcp://127.0.0.1:2375        running  linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Build command

I want to build a multi-platform image for both ARM and AMD and I want to do it in a single buildx command. Additionally I want to store cache for both platfiorms in the same image but with :cache tag.

My image is multi-staging, so I want to push cache for all stages (hence mode=max)

The (simpliified) command to run the build is:

docker buildx build --progress=default --pull --platform linux/amd64,linux/arm64 \
    --cache-from=ghcr.io/potiuk/airflow/main/ci/python3.7:cache \
    --cache to=type=registry,ref=ghcr.io/potiuk/airflow/main/ci/python3.7:cache,mode=max  \
    --push 
    -t ghcr.io/potiuk/airflow/main/ci/python3.7:latest --target main . -f Dockerfile.ci

While the ghcr.io/potiuk/airflow/main/ci/python3.7:latest image is perfectly fine (nice, multiplatform image), the ghcr.io/potiuk/airflow/main/ci/python3.7:cache image only contains cache by the "LAST" build image - i.e if the AMD image was faster to build and push cache, the cache from the ARM builder pushed later seems to override the AMD cache stored there.
I could not find any way to somehow merge those two caches (especially that I cannot specifiy two different cache destination for each of the platforms). This renders the --cache-to,type=registrry essentially useless for multiplatform builds.

I reverted to "inline" mode and it seems to work, but I would really love to keep the latest cache in a separate tag of the image.

Start by reproducing the multi-node build with the shown docker buildx build command, two builders, and a registry cache destination. Inspect how buildx handles --cache-to=type=registry across the linux/amd64 and linux/arm64 nodes. Done means the shared :cache image retains usable cache entries for both platforms rather than only the last node's cache.