EPERM when renaming directory within overlay whose upper layer is tmpfs, from within a nested user namespace
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
### Description
I'm currently hitting `unrecoverable overlayfs inconsistency: failed to make renamed directory opaque: operation not permitted` when running daemonless buildkit, `buildctl-daemonless.sh` as non-root, inside a gVisor pod.
This specifically occurs on any `RUN` step that renames a directory.
This only occurs when using the following gVisor mount hints for a K8s emptydir:
```yaml
dev.gvisor.spec.mount..share: pod
dev.gvisor.spec.mount..type: bind
```
as it results in a gofer-backend lower layer, with a `SelfOverlay` upperlayer. I.e. the container process sees a `tmpfs` mount, which is ultimately backed by the host.
The problem occurs when Buildkit ultimately calls [NeedsUserXAttr()](https://github.com/containerd/containerd/blob/54a25ed63325d3e80f4319cf98a75b4519a8fe70/plugins/snapshots/overlay/overlayutils/check.go#L132), which then makes a call to [IsPathOnTmpfs()](https://github.com/containerd/containerd/blob/54a25ed63325d3e80f4319cf98a75b4519a8fe70/plugins/snapshots/overlay/overlayutils/check.go#L102). This performs a `statfs()` syscall on the directory, which correctly returns the type as `tmpfs`.
As a result, `trusted.overlay.*` is attempted here: https://github.com/google/gvisor/blob/05b57c92aea0bda51ddd1983bf3ca43bee6f089c/pkg/sentry/fsimpl/overlay/filesystem.go#L1322-L1329
which fails because the gofer and sentry processes do not have the required capabilities to set trusted xattrs on the host filesystem.
https://github.com/google/gvisor/pull/12982 resolved this for `9p` upperdirs by adding:
```go
if upperFSName == "9p" && !userXattr {
// Neither the gofer nor sentry processes have the capabilities to set
// trusted xattrs on the host filesystem. Setting `trusted.overlay.*`
// xattrs requires CAP_SYS_ADMIN in the root user namespace. So enable
// `userxattr` mount option as a workaround.
ctx.Infof("overlay.FilesystemType.GetFilesystem: gofer filesystem used as overlay upper layer; enabling userxattr")
userXattr = true
}
```
However my case still fails.
Also for context, the reason I am using these mount hints is because some buildkit builds with `XDG_DATA_HOME` on a gofer mount would lock up and hang indefinitely (similar to https://github.com/google/gvisor/issues/12209). I suspect the mount hints avoided this since it greatly reduced the amount of dentry cache revalidations by the gofer (gvisor dump showed stuck mutexes around dentry cache checks).
### Steps to reproduce
This issue can be reproduced without buildkit:
1. Create a simple pod that uses the pod and bind mount hints on an emptydir:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: test-overlay
annotations:
dev.gvisor.spec.mount.tmp.share: pod
dev.gvisor.spec.mount.tmp.type: bind
spec:
hostUsers: true
runtimeClassName: gvisor
containers:
- name: test
image: ubuntu:latest # or anything with a shell and mount command
args:
- sleep
- 3600
volumeMounts:
- name: tmp
mountPath: /tmp
securityContext:
runAsUser: 1000
volumes:
- name: tmp
emptyDir: {}
```
2. Create the pod and `kubectl exec -it test-overlay -- /bin/bash`
3. Copy and paste the following command. This will trigger a panic in the sandbox and crash the pod:
```sh
unshare -Urm bash -c '
mkdir -p /tmp/{lower,upper,work,merged}
mkdir /tmp/lower/mydir
mount -t overlay overlay -o lowerdir=/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/work /tmp/merged
mv /tmp/merged/mydir /tmp/merged/renamed
'
```
you should see something like:
```sh
waiting on pid 4: waiting on PID 4 in sandbox "db3547cbf88322f52871e6ee1f0760b0b7106710671247cdb35e51d9673955899f7130": urpc method "containerManager.waitPID" failed: EOF
command terminated with exit code 128
```
4. Re-deploy the pod and exec into it again. Copy and paste the following, modified, command. This explicitly defines `userxattr` on the workdir, resulting in a `mv` with no panic:
```sh
unshare -Urm bash -c '
mkdir -p /tmp/{lower,upper,work,merged}
mkdir /tmp/lower/mydir
mount -t overlay overlay -o lowerdir=/tmp/lower,upperdir=/tmp/upper,workdir=/tmp/work,userxattr /tmp/merged
mv /tmp/merged/mydir /tmp/merged/renamed
'
```
### runsc version
```shell
runsc version release-20260727.0
spec: 1.2.1
```
### docker version (if using docker)
```shell
```
### uname
6.12.0-211.39.1.el10_2x86_64
### kubectl (if using Kubernetes)
```shell
K3s: v1.36.0+k3s1
Using Systemd Cgroup driver in both kubelet and containerd / runsc options.
```
### repo state (if built from source)
_No response_
### runsc debug logs (if available)
```shell
```
Contributor guide
Research direction
Start in pkg/sentry/fsimpl/overlay/filesystem.go around the cited overlay setup, and compare the tmpfs upper-layer path with the existing 9p userxattr handling. Reproduce the issue with the provided gVisor pod and unshare commands, then verify that renaming the directory no longer panics without explicitly passing userxattr.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, linux
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100