docker image push stalls on arm64 (raspberry pi3b+)

Description

Pushing an image to ghcr.io does neither succeed nor propagate an error but instead get's stuck with output like

docker image push --all-tags ghcr.io/user/repo/branch
The push refers to repository [ghcr.io/user/repo/branch]
7623dda3f37b: Waiting
5349c496caac: Waiting
ef2ce6ac7bbc: Waiting
87709de4e174: Waiting
7ade7636cc3f: Pushing [=> ] 13.17MB/589.7MB

-> It seems to always fail on the same layer (which is large compared to the others) but not always after the same amounts of bytes -> this varies.

When I cancel the command (Ctrl+C) and retry
-> it shows the exact same output as before (like it tries to resume that)

docker image push --all-tags ghcr.io/user/repo/branch
The push refers to repository [ghcr.io/user/repo/branch]
7623dda3f37b: Waiting
5349c496caac: Waiting
ef2ce6ac7bbc: Waiting
87709de4e174: Waiting
7ade7636cc3f: Pushing [=> ] 13.17MB/589.7MB

--> I need to restart the docker daemon to escape from that

The behaviour is the same when I omit the --all-tags option

These issue are not completely new - I have observed them randomly in the past, but since ~1w it fails all the time

Reproduce

docker image push ghcr.io/user/repo/branch:latest-arm64
docker image push --all-tags ghcr.io/user/repo/branch

Expected behavior

The command should successfully push the image and return

docker version

Client: Docker Engine - Community
 Version:           27.2.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        9e34c9b
 Built:             Fri Sep  6 12:08:03 2024
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.2.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       8b539b8
  Built:            Fri Sep  6 12:08:03 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.21
  GitCommit:        472731909fa34bd7bc9c087e4c27943f9835f111
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    27.2.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 8
 Server Version: 27.2.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.30-v8+
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 907.3MiB
 Name: rpi3bpclient01
 ID: JGMR:RW2V:E2VM:7VKH:4P6Q:S5UT:SE4L:EJSN:MHHQ:FJPH:FI4M:G35A
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 27
  Goroutines: 76
  System Time: 2024-09-12T21:45:21.062689456+02:00
  EventsListeners: 0
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support

Additional Info

Raspberry Pi 3 Model B Plus Rev 1.3

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

uname -srm
Linux 6.6.30-v8+ aarch64

docker logs

Sep 12 22:03:59 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:03:59.273937852+02:00" level=debug msg="Calling POST /v1.47/images/ghcr.io/user/repo/branch/push"
Sep 12 22:03:59 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:03:59.284068275+02:00" level=debug msg="hostDir: /etc/docker/certs.d/ghcr.io"
Sep 12 22:03:59 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:03:59.285969787+02:00" level=debug msg="Trying to push ghcr.io/user/repo/branch to https://ghcr.io"
Sep 12 22:03:59 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:03:59.571015745+02:00" level=debug msg="Pushing repository: ghcr.io/user/repo/branch:build-3d2641a608a4b0258dd3707000454c59f30f81a5-arm64"
Sep 12 22:03:59 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:03:59.576431321+02:00" level=debug msg="Checking for presence of layer sha256:7623dda3f37bd5d32375bce26d736124e775de5fe73cea8f0944ab8655bd71ce (sha256:098640f63054ddbebd>
Sep 12 22:04:00 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:04:00.045461403+02:00" level=debug msg="Checking for presence of layer sha256:5349c496caac60aa08a0ddb3bfee9ccfd383c69d54b7f61c544682790d0f04b0 (sha256:6111518e0cb857b0e7>
Sep 12 22:04:00 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:04:00.277499881+02:00" level=debug msg="Checking for presence of layer sha256:ef2ce6ac7bbc04c0c4d6111bbb8436b4f9fb813a8dad4992108d3d2d862b8e21 (sha256:9039869ca9c26bc53b>
Sep 12 22:04:00 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:04:00.522360032+02:00" level=debug msg="Checking for presence of layer sha256:87709de4e174415c7b14ff9fe3dfdf4cc4d0cd5ed3957c36e5d09501030542d7 (sha256:ec3b03f5a1892a28b5>
Sep 12 22:04:00 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:04:00.753088561+02:00" level=debug msg="Pushing layer: sha256:7ade7636cc3fa2f9b364b3dfbb9d5447391ac0c28497303d19deb5548538187e"
Sep 12 22:04:01 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:04:01.011535179+02:00" level=debug msg="Assembling tar data for l34rbv6fjx8ogvu5kud0f0f21"
Sep 12 22:21:01 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:21:01.291463757+02:00" level=error msg="Upload failed, retrying: write tcp 192.168.0.4:43134->140.82.121.33:443: write: connection timed out"
Sep 12 22:21:06 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:21:06.294928805+02:00" level=debug msg="Pushing layer: sha256:7ade7636cc3fa2f9b364b3dfbb9d5447391ac0c28497303d19deb5548538187e"
Sep 12 22:21:06 rpi3bpclient01 dockerd[40070]: time="2024-09-12T22:21:06.847033571+02:00" level=debug msg="Assembling tar data for l34rbv6fjx8ogvu5kud0f0f21"

The issue names no source file or test. Start by reproducing docker image push on the arm64 Raspberry Pi and inspect the daemon debug logs around “Pushing layer” and “Upload failed, retrying”; done means the push completes or reports a clear error without requiring a daemon restart.