panic: UnlinkAt panics the sandbox when creating a overlay whiteout fails
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
## Description
`unlink(2)` of a copied-up overlay file panics the Sentry and kills the sandbox when creating the matching whiteout fails with `ENOENT`. Multiple panic logs share the same first line:
```text
panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory
```
Crashing goroutine:
```text
panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory
goroutine 42242 gp=0x664741b5860 m=20 mp=0x66473980808 [running]:
panic({0x131c6c0?, 0x6647513cab0?})
GOROOT/src/runtime/panic.go:879 +0x16f fp=0x66474635400 sp=0x66474635350 pc=0x48c94f
gvisor.dev/gvisor/pkg/sentry/fsimpl/overlay.(*filesystem).UnlinkAt(0x66473141200, {0x16afd78, 0x6647a6ccc88}, 0x664745fc488)
pkg/sentry/fsimpl/overlay/filesystem.go:1708 +0xa88 fp=0x66474635758 sp=0x66474635400 pc=0xaf8848
gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).UnlinkAt(0x66473303608, {0x16afd78, 0x6647a6ccc88}, 0x66479d15708?, 0x66473e75d00?)
pkg/sentry/vfs/vfs.go:755 +0x1ad fp=0x664746357f0 sp=0x66474635758 pc=0x8a8a6d
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.unlinkat(0x6647a6ccc88, 0xffffff9c, 0x66474635950?)
pkg/sentry/syscalls/linux/sys_file.go:1137 +0x1a8 fp=0x66474635960 sp=0x664746357f0 pc=0xd02e88
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.Unlink(0x6647a6ccc88?, 0x1f?, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/syscalls/linux/sys_file.go:1124 +0x1d fp=0x66474635988 sp=0x66474635960 pc=0xd02c9d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/kernel/task_syscall.go:143 +0x67d
```
`0xffffff9c` is `AT_FDCWD`; syscall `0x57` is `unlink`. The panic is in `overlay.filesystem.UnlinkAt` after the upper-layer file has been removed and whiteout creation (`mknod` of char `0:0`) returns `ENOENT`.
## Steps to reproduce
Copy-up a lower file under a nested overlay, then race `unlink` of that file (through the overlay) against `rmdir` of its parent on the **raw tmpfs upper** (not through the overlay). After overlay removes the upper name, the directory is empty; `rmdir` can delete the parent before overlay creates the whiteout. Whiteout creation then fails with `ENOENT` and `UnlinkAt` panics.
### 1. runsc (observed)
```shell
docker run --rm --runtime=runsc --cap-add=SYS_ADMIN --entrypoint bash \
gvisor.dev/images/default:latest -c '
set -e
mkdir -p /tmp/l /tmp/uw /tmp/m
mount -t tmpfs tmpfs /tmp/l
mount -t tmpfs tmpfs /tmp/uw
mkdir -p /tmp/uw/upper /tmp/uw/work
mount -t overlay overlay -o lowerdir=/tmp/l,upperdir=/tmp/uw/upper,workdir=/tmp/uw/work /tmp/m
for i in $(seq 0 199); do
mkdir /tmp/l/d$i
echo x > /tmp/l/d$i/f
echo y >> /tmp/m/d$i/f
echo "round $i: unlink /tmp/m/d$i/f"
# Python unlink + rmdir threads: tight syscalls to hit the whiteout race.
python3 -c "
import os, threading
stop = threading.Event()
def spin():
while not stop.is_set():
try: os.rmdir(\"/tmp/uw/upper/d$i\")
except OSError: pass
ts = [threading.Thread(target=spin, daemon=True) for _ in range(4)]
[t.start() for t in ts]
try: os.unlink(\"/tmp/m/d$i/f\")
except OSError as e: print(\"unlink failed but alive:\", e)
stop.set()
for t in ts: t.join()
"
done
echo "no panic after 200 rounds (sandbox still alive)"
'
```
```text
round 27: unlink /tmp/m/d27/f
round 28: unlink /tmp/m/d28/f
round 29: unlink /tmp/m/d29/f
```
The process never prints `sandbox still alive`. `docker run` exits 2. First line of the Sentry panic log is identical to production:
```text
panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory
```
### 2. runc (Linux overlayfs)
Same command with `--runtime=runc`.
```text
round 199: unlink /tmp/m/d199/f
no panic after 200 rounds (sandbox still alive)
```
Exit code 0. The container stays up.
Linux overlayfs does not unlink the upper name and then `mknod` a whiteout. `ovl_cleanup_and_whiteout()` creates the whiteout in the overlay **workdir**, then **renames** it over the upper file (directories use `RENAME_EXCHANGE`). The parent still has that name for the whole operation, so a concurrent `rmdir` of the parent cannot observe an empty directory. If the rename fails, the kernel returns an errno; it does not panic.
gVisor `UnlinkAt` is unlink-then-`mknod`. That is not atomic with respect to other users of the upper filesystem (the overlay `RenameAt` path already notes that essentially no gVisor filesystem supports `RENAME_WHITEOUT`). After the unlink, the upper dir can be empty; `rmdir` on the raw tmpfs upper can remove the parent; whiteout `mknod` then returns `ENOENT`; `UnlinkAt` panics.
## runsc version
```shell
runsc version release-20260824.0-93-gb3674d543f
spec: 1.2.1
```
## docker version (if using docker)
```shell
Client: Docker Engine - Community 26.1.3
Server: Docker Engine - Community 26.1.3
containerd: 1.6.32
```
## uname
`Linux 6.6.110`
## kubectl (if using Kubernetes)
## repo state (if built from source)
`release-20260824.0-93-gb3674d543f`
## runsc debug logs (if available)
Crashing goroutine only:
```shell
panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory
goroutine 42242 gp=0x664741b5860 m=20 mp=0x66473980808 [running]:
panic({0x131c6c0?, 0x6647513cab0?})
GOROOT/src/runtime/panic.go:879 +0x16f fp=0x66474635400 sp=0x66474635350 pc=0x48c94f
gvisor.dev/gvisor/pkg/sentry/fsimpl/overlay.(*filesystem).UnlinkAt(0x66473141200, {0x16afd78, 0x6647a6ccc88}, 0x664745fc488)
pkg/sentry/fsimpl/overlay/filesystem.go:1708 +0xa88 fp=0x66474635758 sp=0x66474635400 pc=0xaf8848
gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).UnlinkAt(0x66473303608, {0x16afd78, 0x6647a6ccc88}, 0x66479d15708?, 0x66473e75d00?)
pkg/sentry/vfs/vfs.go:755 +0x1ad fp=0x664746357f0 sp=0x66474635758 pc=0x8a8a6d
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.unlinkat(0x6647a6ccc88, 0xffffff9c, 0x66474635950?)
pkg/sentry/syscalls/linux/sys_file.go:1137 +0x1a8 fp=0x66474635960 sp=0x664746357f0 pc=0xd02e88
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.Unlink(0x6647a6ccc88?, 0x1f?, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/syscalls/linux/sys_file.go:1124 +0x1d fp=0x66474635988 sp=0x66474635960 pc=0xd02c9d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/kernel/task_syscall.go:143 +0x67d fp=0x66474635cc8 sp=0x66474635988 pc=0xb6913d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallInvoke(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/kernel/task_syscall.go:323 +0x4b fp=0x66474635d20 sp=0x66474635cc8 pc=0xb6a2cb
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallEnter(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/kernel/task_syscall.go:283 +0x68 fp=0x66474635d70 sp=0x66474635d20 pc=0xb69fe8
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscall(0x66474316dc0?)
pkg/sentry/kernel/task_syscall.go:258 +0x2a5 fp=0x66474635e48 sp=0x66474635d70 pc=0xb69d65
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0x664743895c0?, 0x6647a6ccc88)
pkg/sentry/kernel/task_run.go:258 +0xf9c fp=0x66474635f30 sp=0x66474635e48 pc=0xb5e7fc
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0x6647a6ccc88, 0x2b6d)
pkg/sentry/kernel/task_run.go:89 +0x202 fp=0x66474635fc0 sp=0x66474635f30 pc=0xb5d362
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start.gowrap1()
pkg/sentry/kernel/task_start.go:566 +0x1b fp=0x66474635fe0 sp=0x66474635fc0 pc=0xb67c5b
runtime.goexit({})
src/runtime/asm_amd64.s:1771 +0x1 fp=0x66474635fe8 sp=0x66474635fe0 pc=0x495601
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start in goroutine 42119
pkg/sentry/kernel/task_start.go:566 +0xc5
```
Contributor guide
Research direction
Start in pkg/sentry/fsimpl/overlay/filesystem.go at filesystem.UnlinkAt around line 1708, then inspect the overlay RenameAt path and its whiteout handling. Run the supplied nested-overlay reproduction to observe the race. Done means the concurrent unlink/rmdir case no longer panics and reports an appropriate filesystem error while the sandbox remains alive.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100