google / google/gvisor

overlayfs rejects relative lowerdir paths, capping image depth under Docker 29

Open
#14,699 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

gVisor's overlayfs rejects a mount that uses relative `lowerdir` paths. On Linux, the kernel resolves those paths against the working directory of the calling process. The mount succeeds.

Expected: the mount succeeds. Observed: `mount(2)` returns `EINVAL`.

Docker 29 triggers the rejection on images with many layers. containerd switches to relative `lowerdir` paths after a chdir once the absolute option string would pass one page. Shallow images therefore work and deep ones fail:

```
failed to mount /tmp/containerd-mount3312412956: mount source: "overlay",
target: "/tmp/containerd-mount3312412956", fstype: overlay, flags: 0,
data: "workdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/60/work,
upperdir=/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/60/fs,
lowerdir=59/fs:58/fs:57/fs:56/fs:51/fs:...:13/fs", err: invalid argument
```

A tmpfs is already mounted at `/var/lib/docker`, so this is a different failure from #12475.

With the default `--data-root`, images of about 38 layers or more fail. Those images neither run nor build. `docker create paketobuildpacks/builder-jammy-java-tiny` fails at 39 layers. That image is the default builder for Cloud Native Buildpacks. `pack build` and Spring Boot's `bootBuildImage` therefore do not work under gVisor.

### Steps to reproduce

Run this inside a sandbox. Docker is not needed.

```sh
mkdir -p /t/rel && cd /t/rel
mkdir -p l0 l1 u w m

# Relative lowerdir paths. Fails.
mount -t overlay overlay -o lowerdir=l0:l1,upperdir=u,workdir=w m
# mount: mounting overlay on m failed: Invalid argument

# The same mount with absolute paths. Succeeds.
mount -t overlay overlay -o lowerdir=/t/rel/l0:/t/rel/l1,upperdir=/t/rel/u,workdir=/t/rel/w /t/rel/m
```

To separate layer count from path length, I called `mount(2)` directly on a tmpfs and varied each:

| lower layers | paths | option string | result |
| --- | --- | --- | --- |
| 80 | absolute, short | 749 B | mounts |
| 40 | absolute, long | 3119 B | mounts |
| 60 | absolute, long | 4599 B | `ENAMETOOLONG` |
| 10 | relative | about 100 B | `EINVAL` |

The number of lower layers does not cause the failure. The one-page limit on the option string behaves as it does on Linux. Only relative `lowerdir` paths fail.

I verified two workarounds. Both are Docker-specific, and both avoid relative `lowerdir` paths rather than fix the mount. The chdir behaviour lives in containerd's mount helper, so other containerd consumers inside a sandbox are still affected.

- A shorter `--data-root` keeps the absolute string under a page for longer. The limit rises by a few layers.
- `--feature containerd-snapshotter=false` moves Docker off the containerd snapshotter entirely. The classic `overlay2` driver keeps a symlink index at `/l/`, so its option string stays short. The same 39-layer image then creates successfully. #12475 recommends this flag for other Docker 29 problems. It is not a durable answer: Docker documents `overlay2` as "a legacy storage driver that is superseded by the `overlayfs` containerd snapshotter".

### runsc version

GKE Sandbox manages runsc. The node does not expose its version.

### docker version (if using docker)

Docker Engine 29.7.2, in a Docker-in-Docker sidecar. It starts with `--iptables=false --ip6tables=false` and a tmpfs at `/var/lib/docker`.

### uname

`/proc/version` reports `Linux version 4.19.0-gvisor`.

### kubectl (if using Kubernetes)

GKE 1.36.3-gke.1537000. Node pool: `imageType COS_CONTAINERD`, `sandboxConfig.type GVISOR`.

Contributor guide

Open the contributing guide

Research direction

Reproduce the two mount(2) commands inside a sandbox, then trace gVisor's overlayfs handling of the lowerdir mount option. Compare relative and absolute lowerdir resolution while preserving Linux's one-page option-string behavior. Done means relative lowerdir mounts succeed without regressing the documented long-option and absolute-path cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, linux
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.