Swarm repeatedly retries image pull on disk-full nodes, consuming excessive bandwidth

Description

When using Docker Swarm, if a node fails to pull an image due to insufficient disk space, Swarm continuously retries the download in a tight loop. In our case, this behavior resulted in over 2TB of bandwidth consumed in less than 24 hours, with no container ever successfully starting.

This has a major impact on network usage and registry load, especially when image sizes exceed 1GB and failures happen on multiple nodes.

Jun 12 09:00:06 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:06.367754020Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: write /usr/share/kibana/node_modules/@elastic/request-converter>
Jun 12 09:00:06 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:06.369244637Z" level=error msg="pulling image failed" error="failed to register layer: write /usr/share/kibana/node_modules/@elastic/request-converter/dist/schema.json>
Jun 12 09:00:06 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:06.369388754Z" level=error msg="fatal task error" error="No such image: custom_registry.xyz/kibana-oss:latest@sha256:9cae160e0ee294a2a89252096f06acc1ac>
Jun 12 09:00:06 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:06.441383713Z" level=info msg="Layer sha256:1392f8e7c35ede39c33ac7c848af7d868f9fa9429f8b3f114e33fe83ab3c2a1f cleaned up"
Jun 12 09:00:06 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:06.732342895Z" level=warning msg="failed to deactivate service binding for container xyz_kibana.1.62nb9v0qktpxk3o61dxjxgugt" error="No such container: xyzonpr>
Jun 12 09:00:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:10.351369598Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: write /usr/share/elasticsearch/modules/x-pack-ml/platform/linux>
Jun 12 09:00:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:10.352582566Z" level=error msg="pulling image failed" error="failed to register layer: write /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/lib/libmk>
Jun 12 09:00:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:10.352738390Z" level=error msg="fatal task error" error="No such image: custom_registry.xyz/elasticsearch-oss:latest@sha256:e40d111fcf76f521e79cc3c1978>
Jun 12 09:00:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:10.354305414Z" level=info msg="Layer sha256:2e3739e412ea47ca6e8cea1fb174edb6cc9454d57a5c05745fd64111f5481007 cleaned up"
Jun 12 09:00:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:10.354333402Z" level=info msg="Layer sha256:c4793536211cd89f2fa419f22299b223208e4ee90776aa425caf3d45ef6d63bb cleaned up"
Jun 12 09:00:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:10.355872925Z" level=info msg="Layer sha256:7cc14c341faea04af2e1c7b3179c9e782266c27bfe4ad108b0726454ee934e99 cleaned up"
Jun 12 09:00:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:10.365809894Z" level=info msg="Layer sha256:5a67d7de8020dc614f50a93f2123af6a724f36ecfde6ae798404dede73fc2de1 cleaned up"
Jun 12 09:00:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:10.754443579Z" level=warning msg="failed to deactivate service binding for container xyz_elasticsearch.1.rf781l00imknngy9ne6k9xhmy" error="No such container: >
Jun 12 09:00:29 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:29.351123696Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: write /usr/share/elasticsearch/jdk/lib/modules: no space left o>
Jun 12 09:00:29 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:29.352362899Z" level=error msg="pulling image failed" error="failed to register layer: write /usr/share/elasticsearch/jdk/lib/modules: no space left on device" module=>
Jun 12 09:00:29 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:29.352507974Z" level=error msg="fatal task error" error="No such image: custom_registry.xyz/elasticsearch-oss:latest@sha256:e40d111fcf76f521e79cc3c1978>
Jun 12 09:00:29 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:29.352870283Z" level=info msg="Layer sha256:2e3739e412ea47ca6e8cea1fb174edb6cc9454d57a5c05745fd64111f5481007 cleaned up"
Jun 12 09:00:29 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:29.352892650Z" level=info msg="Layer sha256:c4793536211cd89f2fa419f22299b223208e4ee90776aa425caf3d45ef6d63bb cleaned up"
Jun 12 09:00:29 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:29.353942789Z" level=info msg="Layer sha256:7cc14c341faea04af2e1c7b3179c9e782266c27bfe4ad108b0726454ee934e99 cleaned up"
Jun 12 09:00:29 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:29.361776163Z" level=info msg="Layer sha256:5a67d7de8020dc614f50a93f2123af6a724f36ecfde6ae798404dede73fc2de1 cleaned up"
Jun 12 09:00:29 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:29.782420934Z" level=warning msg="failed to deactivate service binding for container xyz_elasticsearch.1.mgy3bmoxqxkxd6ozihgqqt5ck" error="No such container: >
Jun 12 09:00:34 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:34.680761055Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: write /usr/share/kibana/node_modules/@kbn/screenshotting-plugin>
Jun 12 09:00:34 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:34.682130838Z" level=error msg="pulling image failed" error="failed to register layer: write /usr/share/kibana/node_modules/@kbn/screenshotting-plugin/chromium/headles>
Jun 12 09:00:34 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:34.682269499Z" level=error msg="fatal task error" error="No such image: custom_registry.xyz/kibana-oss:latest@sha256:9cae160e0ee294a2a89252096f06acc1ac>
Jun 12 09:00:34 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:34.736480992Z" level=info msg="Layer sha256:1392f8e7c35ede39c33ac7c848af7d868f9fa9429f8b3f114e33fe83ab3c2a1f cleaned up"
Jun 12 09:00:35 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:35.090726402Z" level=warning msg="failed to deactivate service binding for container xyz_kibana.1.ypr5ejllpvwocap5warlgj5yh" error="No such container: xyzonpr>
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.313699548Z" level=error msg="Download failed after 1 attempts: write /var/lib/docker/tmp/GetImageBlob4091921970: no space left on device"
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.429887166Z" level=info msg="Attempting next endpoint for pull after error: write /var/lib/docker/tmp/GetImageBlob4091921970: no space left on device"
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.431869113Z" level=error msg="pulling image failed" error="write /var/lib/docker/tmp/GetImageBlob4091921970: no space left on device" module=node/agent/taskmanager n>
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.432603843Z" level=error msg="fatal task error" error="No such image: custom_registry.xyz/elasticsearch-oss:latest@sha256:e40d111fcf76f521e79cc3c1978>
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.804403917Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: write /usr/share/kibana/node_modules/@kbn/core/packages/http/br>
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.806400283Z" level=error msg="pulling image failed" error="failed to register layer: write /usr/share/kibana/node_modules/@kbn/core/packages/http/browser-internal/sr>
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.806817043Z" level=error msg="fatal task error" error="No such image: custom_registry.xyz/kibana-oss:latest@sha256:9cae160e0ee294a2a89252096f06acc1ac>
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.858391823Z" level=info msg="Layer sha256:1392f8e7c35ede39c33ac7c848af7d868f9fa9429f8b3f114e33fe83ab3c2a1f cleaned up"
Jun 12 09:00:49 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:49.874899512Z" level=warning msg="failed to deactivate service binding for container xyz_elasticsearch.1.nz6b5it3285ugjg3ktkgs5hww" error="No such container: >
Jun 12 09:00:50 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:00:50.250965496Z" level=warning msg="failed to deactivate service binding for container xyz_kibana.1.q31urphimw5adzriafpfc3kr3" error="No such container: xyzonpr>
Jun 12 09:01:09 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:09.739629787Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: write /usr/share/kibana/node_modules/@kbn/esql-ast/src/antlr/es>
Jun 12 09:01:09 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:09.743707971Z" level=error msg="pulling image failed" error="failed to register layer: write /usr/share/kibana/node_modules/@kbn/esql-ast/src/antlr/esql_lexer.js: no s>
Jun 12 09:01:09 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:09.744048446Z" level=error msg="fatal task error" error="No such image: custom_registry.xyz/kibana-oss:latest@sha256:9cae160e0ee294a2a89252096f06acc1ac>
Jun 12 09:01:09 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:09.796583454Z" level=info msg="Layer sha256:1392f8e7c35ede39c33ac7c848af7d868f9fa9429f8b3f114e33fe83ab3c2a1f cleaned up"
Jun 12 09:01:10 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:10.182461632Z" level=warning msg="failed to deactivate service binding for container xyz_kibana.1.66qibvnc32c1j4ixyp4cvfx5t" error="No such container: xyzonpr>
Jun 12 09:01:14 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:14.099756725Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: write /usr/share/elasticsearch/modules/x-pack-core/unboundid-ld>
Jun 12 09:01:14 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:14.100953593Z" level=error msg="pulling image failed" error="failed to register layer: write /usr/share/elasticsearch/modules/x-pack-core/unboundid-ldapsdk-6.0.3.jar: >
Jun 12 09:01:14 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:14.101127972Z" level=error msg="fatal task error" error="No such image: custom_registry.xyz/elasticsearch-oss:latest@sha256:e40d111fcf76f521e79cc3c1978>
Jun 12 09:01:14 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:14.103125817Z" level=info msg="Layer sha256:2e3739e412ea47ca6e8cea1fb174edb6cc9454d57a5c05745fd64111f5481007 cleaned up"
Jun 12 09:01:14 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:14.103227288Z" level=info msg="Layer sha256:c4793536211cd89f2fa419f22299b223208e4ee90776aa425caf3d45ef6d63bb cleaned up"
Jun 12 09:01:14 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:14.104358173Z" level=info msg="Layer sha256:7cc14c341faea04af2e1c7b3179c9e782266c27bfe4ad108b0726454ee934e99 cleaned up"
Jun 12 09:01:14 xyz-dev-vmt dockerd[658124]: time="2025-06-12T09:01:14.113088597Z" level=info msg="Layer sha256:5a67d7de8020dc614f50a93f2123af6a724f36ecfde6ae798404dede73fc2de1 cleaned up"

Expected Behavior

If a node cannot pull an image (e.g., due to disk full), Swarm should:

• Retry with exponential backoff or delay
• Optionally stop retrying after N attempts
• Possibly mark the node as temporarily unschedulable for that service

Steps to Reproduce

• Create a Swarm cluster with at least one node having very low disk space
• Deploy a service with an image that is not yet present on the node
• Observe that the node fails to pull the image
• Swarm retries the pull repeatedly, consuming bandwidth

Workarounds Tried

• Manually labeling nodes to avoid scheduling
• Preloading images manually
• Using restart-policy (not effective since pull happens before container start)

Environment

Client: Docker Engine - Community
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:27 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       45873be
  Built:            Fri May 30 12:07:27 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Client: Docker Engine - Community
 Version:    28.2.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.36.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 22
 Server Version: 28.2.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: active
  NodeID: mk7xr7dh6mdbah6uxs6fwmbqn
  Is Manager: true
  ClusterID: r62w7eooox7fnkal1jq8r046a
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.10.XX.XX
  Manager Addresses:
   10.10.XX.XX:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-60-generic
 Operating System: Ubuntu 24.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.25GiB
 Name: xyz-dev-vmt
 ID: 6910a535-2bb9-4efd-bf9a-1eaea68b43fa
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Would it be possible to:

• Introduce configurable pull retry limits or backoff strategies?
• Provide node-level detection of repeated pull failures and mark nodes as degraded?

Thank you!

Start with the node/agent/taskmanager path identified in the logs and trace how image-pull failures are retried after a no-space-left-on-device error. Reproduce with a Swarm node that has insufficient disk space; done means repeated pulls are delayed or bounded and bandwidth is no longer consumed in a tight loop.