moby / moby/moby

Bloated image unique sizes after update to containerd image store in Docker 29

Open
#51,579 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/images containerd-integration kind/bug
Dominant language
Go
Stars
72.1k
Forks
19.2k
Avg merge
1d 17h
Merged PRs (30d)
164

Description

Description

After upgrading a Docker host with various Docker images to a fresh Docker 29 installation, we have suddendly lost 6 GB of available storage in the /var partition. Investigation shows significant increases in image unique sizes (2x or even 3x).

Rebuilding the images with the "containerd-snapshotter" feature disabled in daemon.json produces again the expected unique sizes, and the images occupy the same as with Docker 28.

Reproduce
  1. Create Dockerfile.base with contents:
FROM oraclelinux:9
RUN dnf -y install openssh-server
  1. Create Dockerfile with contents:
FROM mybase:latest
RUN dnf -y install sudo
  1. Build both images in Docker 29 (fresh installation):
docker build -f Dockerfile-base -t mybase .
docker build -f Dockerfile -t myimage .
  1. Check image sizes with docker system df --verbose:
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
myimage      latest    266eb647e381   20 minutes ago   965MB     576.6MB       388.1MB       0
mybase       latest    d5a5ad4b045d   21 minutes ago   927MB     576.6MB       350.1MB       0
  1. Repeat the steps in either a) Docker 28, b) an existing Docker host upgraded to Docker 29, or c) a fresh Docker 29 installation with the containerd-snapshotter feature disabled. The image sizes are:
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
myimage      latest    8c21b1c4594a   57 minutes ago   589MB     561.5MB       27.74MB       0
mybase       latest    7f911452060c   58 minutes ago   562MB     561.5MB       0B            0
Expected behavior

Image sizes should be similar to those produced by the legacy image store.

docker version
Client: Docker Engine - Community
 Version:           29.0.2
 API version:       1.52
 Go version:        go1.25.4
 Git commit:        8108357
 Built:             Mon Nov 17 12:36:49 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.0.2
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.25.4
  Git commit:       e9ff10b
  Built:            Mon Nov 17 12:33:39 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.1.5
  GitCommit:        fcd43222d6b07379a4be9786bda52438f0dd16a1
 runc:
  Version:          1.3.3
  GitCommit:        v1.3.3-0-gd842d771
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
docker info
Client: Docker Engine - Community
 Version:    29.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.40.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 29.0.2
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: local
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: fcd43222d6b07379a4be9786bda52438f0dd16a1
 runc version: v1.3.3-0-gd842d771
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-314.193.5.4.el9uek.x86_64
 Operating System: Oracle Linux Server 9.6
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.31GiB
 Name: <redacted>
 ID: 162cfa2a-a82e-4245-ae9a-9ab1b05cdbc8
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Registry Mirrors:
  <redacted>
 Live Restore Enabled: false
 Firewall Backend: iptables
Additional Info

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the two-image build with Docker 29 and the containerd-snapshotter feature enabled and disabled, then compare docker system df --verbose output. Trace the image-size and unique-size reporting path involved in the containerd image store. Done means the reproduced images report storage usage comparable to Docker 28 or the legacy image store.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.