`docker manifest push` generates a different digest to what's in the manifest

Description

We create multiarch images and group them with a manifest through docker manifest create, recently our release pipelines started failing because the last step where the manifest is pushed through docker manifest push generates a different digest to what's in the manifest.

Reproduce

• Create 2 docker registries running locally:

docker run --rm -p 5000:5000 --mount type=bind,source=$PWD/registry1,target=/var/lib/registry registry:2
docker run --rm -p 5002:5000 --mount type=bind,source=$PWD/registry2,target=/var/lib/registry registry:2

0. Start a multiarch builder
1. Pull busybox locally, push it to registry1 with a new tag
2. Create a Dockerfile that imports the image pushed to registry1, push it to registry2 with a new tag through docker buildx
3. Create a manifest adding the image in registry2 and push the manifest to registry2

#!/bin/bash

set -x

export DOCKER_CLI_EXPERIMENTAL=enabled

src_registry=localhost:5000
dst_registry=localhost:5002
image_name=busybox
tag=demo

src_full_image_linux_amd64="${src_registry}/${image_name}-linux-amd64:${tag}"
dst_full_image_linux_amd64="${dst_registry}/${image_name}-linux-amd64:${tag}"
dst_full_image="${dst_registry}/${image_name}:${tag}"

# 0. Start a multiarch image
docker run --rm --privileged multiarch/qemu-user-static --reset --credential yes --persistent yes
if ! docker buildx ls | grep -q "multi-arch-builder "; then
docker buildx create --use --name multi-arch-builder --driver-opt network=host
fi

# 1. Push an existing image to src registry
docker pull "${image_name}"
docker tag "${image_name}" "${src_full_image_linux_amd64}"
docker push "${src_full_image_linux_amd64}"

# 2. Create copy in dst registry
echo "FROM ${src_full_image_linux_amd64}" > Dockerfile.copy
docker buildx build . --platform "linux/amd64" \
    --file Dockerfile.copy -t "${dst_full_image_linux_amd64}" --push

# 3. Create a manifest list with a single image and push it to the dst registry
docker manifest rm "${dst_full_image}" 2> /dev/null
docker manifest create --insecure --amend "${dst_full_image}" "${dst_full_image_linux_amd64}"
docker manifest inspect "${dst_full_image}"
docker manifest push --purge "${dst_full_image}"

Expected behavior

The last step should push the manifest digest as it is to registry 2, however the logs say that it has a different digest

docker manifest push --purge localhost:5002/busybox:demo
Pushed ref localhost:5002/busybox@sha256:d50b2da197b84c31511409f0f28b045ab1999259ba62e121e329f3b7a52692e6 with digest: sha256:d3f8c4e9a05f4faeb51191fdb493eea58cd2f27f881324223cbf5dd29d73a6dc
failed to put manifest localhost:5002/busybox:demo: manifest blob unknown: blob unknown to registry

Full logs https://pastebin.com/raw/qK0yg8S1

The error above happens with local registries, with a hosted registry the error is:

+ docker manifest push -p <repo>/<image>:mauricio
Pushed ref <repo>/<image>@sha256:4cc0b3819e74dbb80f23f1a5554b441a8335015d4e4c86561d15b5f22c104c26 with digest: sha256:c516b7a40bd23ad2b4a2f4f79152eda9008b813d0f51145ac2fa0bb39e59a122
Pushed ref <repo>/<image>@sha256:e8974bfd20ea97a6a9413f8fd7124057aedfa5142b16edc394b59692ed322324 with digest: sha256:5ee3d46cdb16f7d326b3f1fffa548b52228c0a21da4e28cc5766d690acd435d7
failed to put manifest <repo>/<image>:mauricio: errors:
manifest invalid: Image sha256:3d23c391ae6c0d2502da44f174023a083951f9afd5718e35253b39b7c89d7d2e references a non-existent child manifest sha256:e8974bfd20ea97a6a9413f8fd7124057aedfa5142b16edc394b59692ed322324
manifest invalid: Image sha256:3d23c391ae6c0d2502da44f174023a083951f9afd5718e35253b39b7c89d7d2e references a non-existent child manifest sha256:4cc0b3819e74dbb80f23f1a5554b441a8335015d4e4c86561d15b5f22c104c26

A successful run prints the same sha, this is a run from previous successful runs:

Step #1 - "build": + docker manifest push -p <img>
Step #1 - "build": Pushed ref <img>@sha256:ff47d66412054a55e62acdbf224a33aba8c019b32cbde4322cee131439898cec with digest: sha256:ff47d66412054a55e62acdbf224a33aba8c019b32cbde4322cee131439898cec
Step #1 - "build": Pushed ref <img>@sha256:644e072bb8434e405fbdfda02b27d9b5b253ba6c482cccc837aa31c4a30489a2 with digest: sha256:644e072bb8434e405fbdfda02b27d9b5b253ba6c482cccc837aa31c4a30489a2
Step #1 - "build": Pushed ref <img>@sha256:4818499c68fcab19d99ba792233a3a794d262dbe4f747a0c857e4328e0c94b6e with digest: sha256:4818499c68fcab19d99ba792233a3a794d262dbe4f747a0c857e4328e0c94b6e
Step #1 - "build": sha256:8f873e79dc5ff72bf609674e23247f19d8e7f0d11f5217f9cd8dbb1853cc694d

docker version

docker version
Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:02:28 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:19 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.12
  GitCommit:        a05d175400b1145e5e6a735a6710579d181e7fb0
 runc:
  Version:          1.0.0+dev
  GitCommit:        v1.0.0-48-g5547b577
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)

Server:
 Containers: 9
  Running: 8
  Paused: 0
  Stopped: 1
 Images: 9
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a05d175400b1145e5e6a735a6710579d181e7fb0
 runc version: v1.0.0-48-g5547b577
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.19.11-1rodete1-amd64
 Operating System: Debian GNU/Linux rodete
 OSType: linux
 Architecture: x86_64
 ...
 Debug Mode: true
  File Descriptors: 86
  Goroutines: 88
  System Time: 2023-01-14T01:18:19.511794451Z
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.gcr.io/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.168.10.0/22, Size: 24

docker buildx version

docker buildx version
github.com/docker/buildx v0.9.1-docker ed00243a0ce2a0aee75311b06e32d33b44729689


docker buildx inspect multi-arch-builder
Name:   multi-arch-builder
Driver: docker-container

Nodes:
Name:           multi-arch-builder0
Endpoint:       unix:///var/run/docker.sock
Driver Options: network="host"
Status:         running
Buildkit:       v0.11.0
Platforms:      linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Start by running the provided two-registry reproduction with docker manifest create and docker manifest push, using the reported Docker and buildx versions. Trace the CLI path for docker manifest push and compare the locally calculated manifest and child digests with the registry responses; done means the pushed manifest preserves the digest and references valid child manifests.