docker-ce 29.0.0 with nested overlayfs storage driver seems not support whiteout (deleting file on lower layer) in improper dind setup
Open
Nobody has claimed this yet.
kind/bug
status/0-triage
version/29.0
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 2.2k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 43
Description
Description
In docker-ce_29.0.0-1~debian.11~bullseye_amd64.deb, Cannot build or load a image with following Dockerfile:
FROM alpine
RUN rm /etc/hostname
the problem is: docker-ce 29.0.0-1 cannot delete files in lower layer.
Reproduce
Run following commands in docker run --rm -it --privileged debian:bullseye (docker in docker setup)
- apt-get update && apt-get install -y ca-certificates curl
- install -m 0755 -d /etc/apt/keyrings
- curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
- chmod a+r /etc/apt/keyrings/docker.asc
- tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: bullseye
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF - apt-get update
- curl -O https://download.docker.com/linux/debian/dists/bullseye/pool/stable/amd64/docker-ce_29.0.0-1~debian.11~bullseye_amd64.deb
- apt-get install -y docker-ce_29.0.0-1~debian.11~bullseye_amd64.deb
- dockerd &
- mkdir whiteout && cd whiteout
- (echo "FROM alpine"; echo "RUN rm /etc/hostname") | tee Dockerfile
- docker build .
result:
[+] Building 0.9s (5/5) docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 70B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.8s
[+] Building 1.0s (5/5) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 70B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.8s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0sv
=> CACHED [1/2] FROM docker.io/library/alpine:latest@sha256:4b7ce07002c69e8f3d704a9c5d6fd3053be500b7f1c69fc0d80990c2ad8dd412 0.0sy
=> => resolve docker.io/library/alpine:latest@sha256:4b7ce07002c69e8f3d704a9c5d6fd3053be500b7f1c69fc0d80990c2ad8dd412 0.0sf
=> ERROR [2/2] RUN rm /etc/hostname 0.1s
------
> [2/2] RUN rm /etc/hostname:
------
Dockerfile:2
--------------------
1 | FROM alpine
2 | >>> RUN rm /etc/hostname
3 |
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c rm /etc/hostname" did not complete successfully: mount source: "overlay", target: "/var/lib/docker/buildkit/containerd-overlayfs/cachemounts/buildkit4108886608", fstype: overlay, flags: 0, data: "workdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/7/work,upperdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs,lowerdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/5/fs,index=off,redirect_dir=off", err: invalid argument
Expected behavior
Can build the image with 28.5.2-1~debian.11~bullseye
[+] Building 3.3s (7/7) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 70B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 2.2s
=> [auth] library/alpine:pull token for registry-1.docker.io 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/2] FROM docker.io/library/alpine:latest@sha256:4b7ce07002c69e8f3d704a9c5d6fd3053be500b7f1c69fc0d80990c2ad8dd412 0.5s
=> => resolve docker.io/library/alpine:latest@sha256:4b7ce07002c69e8f3d704a9c5d6fd3053be500b7f1c69fc0d80990c2ad8dd412 0.0s
=> => sha256:4b7ce07002c69e8f3d704a9c5d6fd3053be500b7f1c69fc0d80990c2ad8dd412 9.22kB / 9.22kB 0.0s
=> => sha256:85f2b723e106c34644cd5851d7e81ee87da98ac54672b29947c052a45d31dc2f 1.02kB / 1.02kB 0.0s
=> => sha256:706db57fb2063f39f69632c5b5c9c439633fda35110e65587c5d85553fd1cc38 581B / 581B 0.0s
=> => sha256:2d35ebdb57d9971fea0cac1582aa78935adf8058b2cc32db163c98822e5dfa1b 3.80MB / 3.80MB 0.3s
=> => extracting sha256:2d35ebdb57d9971fea0cac1582aa78935adf8058b2cc32db163c98822e5dfa1b 0.1s
=> [2/2] RUN rm /etc/hostname 0.2s
=> exporting to image 0.2s
=> => exporting layers 0.1s
=> => writing image sha256:321c98552e5e5038b745530ffb8dd25ab3e297d59fabca4956520a7a9967aee8 0.0s
=> => naming to docker.io/library/whiteout
docker version
Client: Docker Engine - Community
Version: 29.0.0
API version: 1.52
Go version: go1.25.4
Git commit: 3d4129b
Built: Mon Nov 10 21:47:13 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 29.0.0
API version: 1.52 (minimum version 1.44)
Go version: go1.25.4
Git commit: d105562
Built: Mon Nov 10 21:47:13 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v2.1.5
GitCommit: fcd43222d6b07379a4be9786bda52438f0dd16a1
runc:
Version: 1.3.3
GitCommit: v1.3.3-0-gd842d771
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client: Docker Engine - Community
Version: 29.0.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.29.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.40.3
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 29.0.0
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: fcd43222d6b07379a4be9786bda52438f0dd16a1
runc version: v1.3.3-0-gd842d771
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.17.7-arch1-1
Operating System: Debian GNU/Linux 11 (bullseye)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 30.96GiB
Name: eb09c1a85cc1
ID: 8cad55b5-9fc1-47e4-97b8-2d7eeec169f8
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Firewall Backend: iptables
Additional Info
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure in the nested Docker-in-Docker setup described in the issue, using the two-line Dockerfile with FROM alpine and RUN rm /etc/hostname. Compare Docker Engine 29.0.0 with 28.5.2 and inspect the overlayfs configuration shown in the error. Done means the image builds successfully with the nested overlayfs setup, or the regression and affected component are identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- debian, docker, linux
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100