google / google/gvisor

EPERM when renaming directory within overlay whose upper layer is tmpfs, from within a nested user namespace

Open
#14,033 14 comments 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

I'm currently hitting `unrecoverable overlayfs inconsistency: failed to make renamed directory opaque: operation not permitted` when running daemonless buildkit, `buildctl-daemonless.sh` as non-root, inside a gVisor pod.
This specifically occurs on any `RUN` step that renames a directory.

This only occurs when using the following gVisor mount hints for a K8s emptydir:
```yaml
dev.gvisor.spec.mount..share: pod
dev.gvisor.spec.mount..type: bind
```
as it results in a gofer-backend lower layer, with a `SelfOverlay` upperlayer. I.e. the container process sees a `tmpfs` mount, which is ultimately backed by the host.

The problem occurs when Buildkit ultimately calls [NeedsUserXAttr()](https://github.com/containerd/containerd/blob/54a25ed63325d3e80f4319cf98a75b4519a8fe70/plugins/snapshots/overlay/overlayutils/check.go#L132), which then makes a call to [IsPathOnTmpfs()](https://github.com/containerd/containerd/blob/54a25ed63325d3e80f4319cf98a75b4519a8fe70/plugins/snapshots/overlay/overlayutils/check.go#L102). This performs a `statfs()` syscall on the directory, which correctly returns the type as `tmpfs`.

As a result, `trusted.overlay.*` is attempted here: https://github.com/google/gvisor/blob/05b57c92aea0bda51ddd1983bf3ca43bee6f089c/pkg/sentry/fsimpl/overlay/filesystem.go#L1322-L1329

which fails because the gofer and sentry processes do not have the required capabilities to set trusted xattrs on the host filesystem.

https://github.com/google/gvisor/pull/12982 resolved this for `9p` upperdirs by adding:
```go
if upperFSName == "9p" && !userXattr {
// Neither the gofer nor sentry processes have the capabilities to set
// trusted xattrs on the host filesystem. Setting `trusted.overlay.*`
// xattrs requires CAP_SYS_ADMIN in the root user namespace. So enable
// `userxattr` mount option as a workaround.
ctx.Infof("overlay.FilesystemType.GetFilesystem: gofer filesystem used as overlay upper layer; enabling userxattr")
userXattr = true
}
```
However my case still fails.

Also for context, the reason I am using these mount hints is because some buildkit builds with `XDG_DATA_HOME` on a gofer mount would lock up and hang indefinitely (similar to https://github.com/google/gvisor/issues/12209). I suspect the mount hints avoided this since it greatly reduced the amount of dentry cache revalidations by the gofer (gvisor dump showed stuck mutexes around dentry cache checks).

### Steps to reproduce

This issue can be reproduced without buildkit:

1. Create a simple pod that uses the pod and bind mount hints on an emptydir:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: test-overlay
annotations:
dev.gvisor.spec.mount.tmp.share: pod
dev.gvisor.spec.mount.tmp.type: bind
spec:
hostUsers: true
runtimeClassName: gvisor
containers:
- name: test
image: ubuntu:latest # or anything with a shell and mount command
args:
- sleep
- 3600
volumeMounts:
- name: tmp
mountPath: /tmp
securityContext:
runAsUser: 1000
volumes:
- name: tmp
emptyDir: {}
```
2. Create the pod and `kubectl exec -it test-overlay -- /bin/bash`
3. Copy and paste the following command. This will trigger a panic in the sandbox and crash the pod:
```sh
unshare -Urm bash -c '
mkdir -p /tmp/{lower,upper,work,merged}
mkdir /tmp/lower/mydir
mount -t overlay overlay -o lowerdir=/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/work /tmp/merged
mv /tmp/merged/mydir /tmp/merged/renamed
'
```
you should see something like:
```sh
waiting on pid 4: waiting on PID 4 in sandbox "db3547cbf88322f52871e6ee1f0760b0b7106710671247cdb35e51d9673955899f7130": urpc method "containerManager.waitPID" failed: EOF
command terminated with exit code 128
```
4. Re-deploy the pod and exec into it again. Copy and paste the following, modified, command. This explicitly defines `userxattr` on the workdir, resulting in a `mv` with no panic:
```sh
unshare -Urm bash -c '
mkdir -p /tmp/{lower,upper,work,merged}
mkdir /tmp/lower/mydir
mount -t overlay overlay -o lowerdir=/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/work,userxattr /tmp/merged
mv /tmp/merged/mydir /tmp/merged/renamed
'
```

### runsc version

```shell
runsc version release-20260727.0
spec: 1.2.1
```

### docker version (if using docker)

```shell

```

### uname

6.12.0-211.39.1.el10_2x86_64

### kubectl (if using Kubernetes)

```shell
K3s: v1.36.0+k3s1

Using Systemd Cgroup driver in both kubelet and containerd / runsc options.
```

### repo state (if built from source)

_No response_

### runsc debug logs (if available)

```shell

```

Contributor guide

Open the contributing guide

Research direction

Start in pkg/sentry/fsimpl/overlay/filesystem.go around the cited overlay setup, and compare the tmpfs upper-layer path with the existing 9p userxattr handling. Reproduce the issue with the provided gVisor pod and unshare commands, then verify that renaming the directory no longer panics without explicitly passing userxattr.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, linux
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.