Pushing large image fails with Gateway errors

In my multi-arch build of a larger image, I get errors when pushing to the image registry. For quay.io, I get a 502 Bad Gateway error and for docker.io I get a 504 Gateway Time-out error.

How can I resolve those?

Steps to reproduce

• I'm building a Jupyter Lab image base on this code: https://github.com/lehrig/kubeflow-ppc64le-notebook-images
• I basically followed this comment https://github.com/docker/buildx/issues/327#issuecomment-665958307 to save the time to rebuild for every test; the same issues occur when I directly build & push.

Environment configuration:

git clone https://github.com/lehrig/kubeflow-ppc64le-notebook-images
cd kubeflow-ppc64le-notebook-images

export ELYRA_VERSION=3.11.1
export PYTHON_VERSION=3.8
export TENSORFLOW_VERSION=2.8.1
export SUPPORT_GPU=true
export MINOR_RELEASE=0

export IMAGE=quay.io/ibm/kubeflow-notebook-image-ppc64le
export TAG=elyra${ELYRA_VERSION}-py${PYTHON_VERSION}-tf${TENSORFLOW_VERSION}-v${MINOR_RELEASE}
export TARGET=$IMAGE:$TAG

For building into the cache:

docker build --build-arg NB_GID=0 --build-arg ELYRA_VERSION=$ELYRA_VERSION --build-arg PYTHON_VERSION=$PYTHON_VERSION --build-arg TENSORFLOW_VERSION=$TENSORFLOW_VERSION --build-arg SUPPORT_GPU=$SUPPORT_GPU -t $TARGET -f Dockerfile --platform linux/amd64,linux/ppc64le --cache-to=type=local,dest=cache,mode=max .

For pushing to quay.io:

docker build --build-arg NB_GID=0 --build-arg ELYRA_VERSION=$ELYRA_VERSION --build-arg PYTHON_VERSION=$PYTHON_VERSION --build-arg TENSORFLOW_VERSION=$TENSORFLOW_VERSION --build-arg SUPPORT_GPU=$SUPPORT_GPU -t $TARGET -f Dockerfile --platform linux/amd64,linux/ppc64le --push --cache-from=type=local,src=cache .  

For pushing to docker.io:

docker build --build-arg NB_GID=0 --build-arg ELYRA_VERSION=$ELYRA_VERSION --build-arg PYTHON_VERSION=$PYTHON_VERSION --build-arg TENSORFLOW_VERSION=$TENSORFLOW_VERSION --build-arg SUPPORT_GPU=$SUPPORT_GPU -t docker.io/lehrig/kubeflow-notebook-image-ppc64le:elyra3.11.1-py3.8-tf2.8.1-v0 -f Dockerfile --platform linux/amd64,linux/ppc64le --push --cache-from=type=local,src=cache .

Results

Case 1: pushing to quay.io

 => [linux/ppc64le 11/11] WORKDIR /home/jovyan                                                                                                                                                                                                                          75.6s
 => => sha256:dced7e77e0c296412c7e8ccd6e64470b70c070b2a1d894e198369b9c575ff75b 3.82GB / 3.82GB                                                                                                                                                                          75.3s
 => exporting to image                                                                                                                                                                                                                                                 143.5s
 => => # error: failed to copy: unexpected status: 502 Bad Gateway                                                                                                                                                                                                           
 => => # retrying in 1s 

Case 2: pushing to docker.io

 => [linux/ppc64le 11/11] WORKDIR /home/jovyan                                                                                                                                                                                                                          65.4s
 => => sha256:dced7e77e0c296412c7e8ccd6e64470b70c070b2a1d894e198369b9c575ff75b 3.82GB / 3.82GB    
 => exporting to image                                                                                                                                                                                                                                                  78.1s
 => => # error: failed to copy: unexpected status: 504 Gateway Time-out                                                                                                                                                                                                      
 => => # retrying in 1s

Environment

• Docker version 20.10.17, build 100c701
• github.com/docker/buildx v0.9.1 ed00243a0ce2a0aee75311b06e32d33b44729689
• MacOS 12.2.1 (Montery)

Start by reproducing the multi-architecture push with Docker 20.10.17, buildx v0.9.1, and the commands and image configuration in the issue. Trace the buildx image export and registry push path while comparing the 502 and 504 responses from quay.io and docker.io. Done means identifying a buildx-side cause or a confirmed registry limitation and documenting the resolution.