overlayfs rejects relative lowerdir paths, capping image depth under Docker 29
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
### Description
gVisor's overlayfs rejects a mount that uses relative `lowerdir` paths. On Linux, the kernel resolves those paths against the working directory of the calling process. The mount succeeds.
Expected: the mount succeeds. Observed: `mount(2)` returns `EINVAL`.
Docker 29 triggers the rejection on images with many layers. containerd switches to relative `lowerdir` paths after a chdir once the absolute option string would pass one page. Shallow images therefore work and deep ones fail:
```
failed to mount /tmp/containerd-mount3312412956: mount source: "overlay",
target: "/tmp/containerd-mount3312412956", fstype: overlay, flags: 0,
data: "workdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/60/work,
upperdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/60/fs,
lowerdir=59/fs:58/fs:57/fs:56/fs:51/fs:...:13/fs", err: invalid argument
```
A tmpfs is already mounted at `/var/lib/docker`, so this is a different failure from #12475.
With the default `--data-root`, images of about 38 layers or more fail. Those images neither run nor build. `docker create paketobuildpacks/builder-jammy-java-tiny` fails at 39 layers. That image is the default builder for Cloud Native Buildpacks. `pack build` and Spring Boot's `bootBuildImage` therefore do not work under gVisor.
### Steps to reproduce
Run this inside a sandbox. Docker is not needed.
```sh
mkdir -p /t/rel && cd /t/rel
mkdir -p l0 l1 u w m
# Relative lowerdir paths. Fails.
mount -t overlay overlay -o lowerdir=l0:l1,upperdir=u,workdir=w m
# mount: mounting overlay on m failed: Invalid argument
# The same mount with absolute paths. Succeeds.
mount -t overlay overlay -o lowerdir=/t/rel/l0:/t/rel/l1,upperdir=/t/rel/u,workdir=/t/rel/w /t/rel/m
```
To separate layer count from path length, I called `mount(2)` directly on a tmpfs and varied each:
| lower layers | paths | option string | result |
| --- | --- | --- | --- |
| 80 | absolute, short | 749 B | mounts |
| 40 | absolute, long | 3119 B | mounts |
| 60 | absolute, long | 4599 B | `ENAMETOOLONG` |
| 10 | relative | about 100 B | `EINVAL` |
The number of lower layers does not cause the failure. The one-page limit on the option string behaves as it does on Linux. Only relative `lowerdir` paths fail.
I verified two workarounds. Both are Docker-specific, and both avoid relative `lowerdir` paths rather than fix the mount. The chdir behaviour lives in containerd's mount helper, so other containerd consumers inside a sandbox are still affected.
- A shorter `--data-root` keeps the absolute string under a page for longer. The limit rises by a few layers.
- `--feature containerd-snapshotter=false` moves Docker off the containerd snapshotter entirely. The classic `overlay2` driver keeps a symlink index at `/l/`, so its option string stays short. The same 39-layer image then creates successfully. #12475 recommends this flag for other Docker 29 problems. It is not a durable answer: Docker documents `overlay2` as "a legacy storage driver that is superseded by the `overlayfs` containerd snapshotter".
### runsc version
GKE Sandbox manages runsc. The node does not expose its version.
### docker version (if using docker)
Docker Engine 29.7.2, in a Docker-in-Docker sidecar. It starts with `--iptables=false --ip6tables=false` and a tmpfs at `/var/lib/docker`.
### uname
`/proc/version` reports `Linux version 4.19.0-gvisor`.
### kubectl (if using Kubernetes)
GKE 1.36.3-gke.1537000. Node pool: `imageType COS_CONTAINERD`, `sandboxConfig.type GVISOR`.
Contributor guide
Research direction
Reproduce the two mount(2) commands inside a sandbox, then trace gVisor's overlayfs handling of the lowerdir mount option. Compare relative and absolute lowerdir resolution while preserving Linux's one-page option-string behavior. Done means relative lowerdir mounts succeed without regressing the documented long-option and absolute-path cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, linux
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100