dockerd ↔ containerd exchange ~6.8 MB/s of image metadata while the daemon is completely idle (containerd image store)

Description

On an idle host, dockerd and containerd together consume ~2 of 4 CPU cores
continuously. All of it is caused by a sustained, unthrottled data exchange over
/run/containerd/containerd.sock: ~6.8 MB/s, ~4,500 writes/s, with no
containers being created, no external API clients, and no meaningful log output.

This has been running continuously for 26 days of uptime, across two Engine
versions and two containerd versions.

The payload is image metadata. The load scales with the number of images: removing
69 dangling images and 394 build cache entries reduced total CPU from 223% to 182%,
and a second cleanup (10 unused images) took it from 191% to 164%. Nothing else
ever helped — not a full daemon restart, not upgrading either component.

Reproduce

This is not a deliberate reproduction — it is the steady state of a production host.

1. Run Docker 29.x with the containerd image store (default since v29) on a host
   with a non-trivial number of images (28 images / 658 content blobs here) and
   containers (37 running here).

2. Leave the host idle: no builds, no pulls, no deploys, no external API clients.

3. Find containerd's end of the socket:

   ss -xp | grep containerd.sock

4. Measure the traffic on that fd:

   CPID=$(pgrep -x containerd)
   bpftrace -e "tracepoint:syscalls:sys_enter_write /pid==$CPID && args->fd==13/
   { @bytes = sum(args->count); @n = count(); }
   interval:s:5 { exit(); }"

5. Observe ~6.8 MB/s and ~4,500 writes/s, continuously, while nothing is happening.

Expected behavior

An idle daemon should exchange approximately nothing with containerd.

Observed instead: 6.8 MB/s sustained — roughly 590 GB/day between two processes on
the same machine — consuming ~2 of 4 CPU cores and causing 69% of the host's
context switches.

docker version

Client: Docker Engine - Community
 Version:           29.7.2
 API version:       1.55
 Go version:        go1.26.5
 Git commit:        a7dcaa6
 Built:             Wed Aug  5 18:28:53 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.7.2
  API version:      1.55 (minimum version 1.40)
  Go version:       go1.26.5
  Git commit:       6a43e3d
  Built:            Wed Aug  5 18:28:53 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.3.4
  GitCommit:        db8809540e1a7a9da5d518876894933ff55692ab
 runc:
  Version:          1.5.1
  GitCommit:        v1.5.1-0-g8f2685a4
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    29.7.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.) v0.35.0
  compose: Docker Compose (Docker Inc.) v5.3.1

Server:
 Containers: 38
  Running: 37
  Paused: 0
  Stopped: 1
 Images: 28
 Server Version: 29.7.2
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: db8809540e1a7a9da5d518876894933ff55692ab
 runc version: v1.5.1-0-g8f2685a4
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-137-generic
 Operating System: Ubuntu 24.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.755GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Live Restore Enabled: false
 Firewall Backend: iptables
  EnableUserlandProxy: true

Additional Info

Measurements

CPU (5 s samples, mean of 10):

containerd   ~96 % of one core
dockerd      ~94 % of one core
split        ~80 % user time, ~20 % system time

Context switches: dockerd 16,185/s, containerd 16,021/s — together 32,206 of
the system's 46,901/s (69 %), almost entirely voluntary.

Syscalls per process: ~48,000/s.

epoll_pwait  ~13,400/s
futex         ~8,600/s
nanosleep     ~5,800/s
read          ~5,700/s
write         ~5,100/s

A single fd pair carries essentially all of it. Per-fd breakdown over 5 s:

dockerd fd 9   27,105 reads + 24,369 writes    <- everything else < 100
dockerd fd 7    1,803 +  1,798                 (Go runtime eventfd)

ss -xp confirms the peer:

/run/containerd/containerd.sock
  dockerd fd 9  <-->  containerd fd 13

Throughput: 33,824,204 bytes in 5 s over 22,470 writes = 6.8 MB/s.

It is continuous, not periodic. Bytes/second over 15 s:

7.69  7.31  5.90  8.41  5.92  8.20  5.93  8.20  6.33
7.42  7.84  7.09  6.31  7.78  6.63   (MB)

The payload is image metadata. Readable fragments captured from the write
buffers include mediaType, //github.com/cirruslabs/docker-images-flutter (the
org.opencontainers.image.source label of an image present on the host), package
names such as imagemagick and docker-cli-compose, and content digests.

perf profile of dockerd is dominated by Go GC — and is unchanged whether the
heap is 696 MB or 121 MB:

7.56 %  runtime.tryDeferToSpanScan
3.52 %  runtime.mallocgcSmallScanNoHeader
3.15 %  runtime.memclrNoHeapPointers
3.03 %  runtime.scanObjectsSmall

Goroutine dumps (SIGUSR1) show nothing spinning: dockerd has 279 goroutines,
almost all waiting (148 IO wait, 72 sync.Cond.Wait — 70 of those are the
per-container fifo/bytespipe log pipeline). containerd has 165, with two
streamNamespaceInterceptor goroutines, i.e. two open streams.

The load scales with the number of images

This is the strongest hint at the cause.

action	total CPU
baseline (98 images, 450 build cache entries)	223 %
after docker image prune + builder prune (→ 29 images)	182 %
after full systemctl restart docker	199 %
after containerd.io 2.2.6 → 2.3.4	187 %
after docker-ce 29.6.2 → 29.7.2	191 %
after removing 10 more unused images (→ 27)	164 %

Only image removal ever helped, and it helped twice, in proportion.

Ruled out

hypothesis	ruled out by
container churn	0 container events in 180 s
external API clients	only 2 connections to docker.sock: dockerd itself and systemd socket activation
health checks	16 exec cycles/minute — three orders of magnitude too few
log processing	container logs grow ~1 KB per 5 s; journald 140 lines/min
cgroup stat collection	would be system time; 80 % is user time
memory leak	RSS stable (+4 KB in 60 s)
accumulated daemon state	full restart dropped RSS 696 → 121 MB, CPU unchanged
containerd version	2.2.6 → 2.3.4, no change
engine version	29.6.2 → 29.7.2, no change

Commands used

# per-fd read/write breakdown
bpftrace -e 'tracepoint:syscalls:sys_enter_write /comm=="dockerd"/ { @w[args->fd] = count(); }
             tracepoint:syscalls:sys_enter_read  /comm=="dockerd"/ { @r[args->fd] = count(); }
             interval:s:5 { exit(); }'

# identify the peer
readlink /proc/$(pidof dockerd)/fd/9      # -> socket:[INODE]
ss -xp | grep INODE

# throughput
bpftrace -e 'tracepoint:syscalls:sys_enter_write /pid==CONTAINERD_PID && args->fd==13/
             { @bytes = sum(args->count); @n = count(); } interval:s:5 { exit(); }'

# per-second profile (shows it is continuous, not periodic)
bpftrace -e 'tracepoint:syscalls:sys_enter_write /pid==CONTAINERD_PID && args->fd==13/
             { @b = sum(args->count); } interval:s:1 { print(@b); clear(@b); }'

# payload sampling
bpftrace -e 'tracepoint:syscalls:sys_enter_write /pid==CONTAINERD_PID && args->fd==13 && args->count>200/
             { printf("%s~", str(args->buf, 180)); }'

Notes

• Not reproduced deliberately on a second host; observed continuously on this one
  for 26 days across two version upgrades.
• Not tested with the classic (graph driver) image store — the host has not been
  switched back.
• Hostname and Docker ID removed from the docker info output above.

Start by tracing the dockerd/containerd socket with the listed bpftrace commands and reviewing the image-metadata payload and per-fd measurements. Done means identifying the source of the continuous idle exchange and stopping the sustained metadata traffic, CPU use, and context switching without affecting normal image operations.