google / google/gvisor

panic: UnlinkAt panics the sandbox when creating a overlay whiteout fails

Open
#14,726 3 comments 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

## Description

`unlink(2)` of a copied-up overlay file panics the Sentry and kills the sandbox when creating the matching whiteout fails with `ENOENT`. Multiple panic logs share the same first line:

```text
panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory
```

Crashing goroutine:

```text
panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory

goroutine 42242 gp=0x664741b5860 m=20 mp=0x66473980808 [running]:
panic({0x131c6c0?, 0x6647513cab0?})
GOROOT/src/runtime/panic.go:879 +0x16f fp=0x66474635400 sp=0x66474635350 pc=0x48c94f
gvisor.dev/gvisor/pkg/sentry/fsimpl/overlay.(*filesystem).UnlinkAt(0x66473141200, {0x16afd78, 0x6647a6ccc88}, 0x664745fc488)
pkg/sentry/fsimpl/overlay/filesystem.go:1708 +0xa88 fp=0x66474635758 sp=0x66474635400 pc=0xaf8848
gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).UnlinkAt(0x66473303608, {0x16afd78, 0x6647a6ccc88}, 0x66479d15708?, 0x66473e75d00?)
pkg/sentry/vfs/vfs.go:755 +0x1ad fp=0x664746357f0 sp=0x66474635758 pc=0x8a8a6d
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.unlinkat(0x6647a6ccc88, 0xffffff9c, 0x66474635950?)
pkg/sentry/syscalls/linux/sys_file.go:1137 +0x1a8 fp=0x66474635960 sp=0x664746357f0 pc=0xd02e88
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.Unlink(0x6647a6ccc88?, 0x1f?, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/syscalls/linux/sys_file.go:1124 +0x1d fp=0x66474635988 sp=0x66474635960 pc=0xd02c9d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/kernel/task_syscall.go:143 +0x67d
```

`0xffffff9c` is `AT_FDCWD`; syscall `0x57` is `unlink`. The panic is in `overlay.filesystem.UnlinkAt` after the upper-layer file has been removed and whiteout creation (`mknod` of char `0:0`) returns `ENOENT`.

## Steps to reproduce

Copy-up a lower file under a nested overlay, then race `unlink` of that file (through the overlay) against `rmdir` of its parent on the **raw tmpfs upper** (not through the overlay). After overlay removes the upper name, the directory is empty; `rmdir` can delete the parent before overlay creates the whiteout. Whiteout creation then fails with `ENOENT` and `UnlinkAt` panics.

### 1. runsc (observed)

```shell
docker run --rm --runtime=runsc --cap-add=SYS_ADMIN --entrypoint bash \
gvisor.dev/images/default:latest -c '
set -e
mkdir -p /tmp/l /tmp/uw /tmp/m
mount -t tmpfs tmpfs /tmp/l
mount -t tmpfs tmpfs /tmp/uw
mkdir -p /tmp/uw/upper /tmp/uw/work
mount -t overlay overlay -o lowerdir=/tmp/l,upperdir=/tmp/uw/upper,workdir=/tmp/uw/work /tmp/m
for i in $(seq 0 199); do
mkdir /tmp/l/d$i
echo x > /tmp/l/d$i/f
echo y >> /tmp/m/d$i/f
echo "round $i: unlink /tmp/m/d$i/f"
# Python unlink + rmdir threads: tight syscalls to hit the whiteout race.
python3 -c "
import os, threading
stop = threading.Event()
def spin():
while not stop.is_set():
try: os.rmdir(\"/tmp/uw/upper/d$i\")
except OSError: pass
ts = [threading.Thread(target=spin, daemon=True) for _ in range(4)]
[t.start() for t in ts]
try: os.unlink(\"/tmp/m/d$i/f\")
except OSError as e: print(\"unlink failed but alive:\", e)
stop.set()
for t in ts: t.join()
"
done
echo "no panic after 200 rounds (sandbox still alive)"
'
```

```text
round 27: unlink /tmp/m/d27/f
round 28: unlink /tmp/m/d28/f
round 29: unlink /tmp/m/d29/f
```

The process never prints `sandbox still alive`. `docker run` exits 2. First line of the Sentry panic log is identical to production:

```text
panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory
```

### 2. runc (Linux overlayfs)

Same command with `--runtime=runc`.

```text
round 199: unlink /tmp/m/d199/f
no panic after 200 rounds (sandbox still alive)
```

Exit code 0. The container stays up.

Linux overlayfs does not unlink the upper name and then `mknod` a whiteout. `ovl_cleanup_and_whiteout()` creates the whiteout in the overlay **workdir**, then **renames** it over the upper file (directories use `RENAME_EXCHANGE`). The parent still has that name for the whole operation, so a concurrent `rmdir` of the parent cannot observe an empty directory. If the rename fails, the kernel returns an errno; it does not panic.

gVisor `UnlinkAt` is unlink-then-`mknod`. That is not atomic with respect to other users of the upper filesystem (the overlay `RenameAt` path already notes that essentially no gVisor filesystem supports `RENAME_WHITEOUT`). After the unlink, the upper dir can be empty; `rmdir` on the raw tmpfs upper can remove the parent; whiteout `mknod` then returns `ENOENT`; `UnlinkAt` panics.

## runsc version

```shell
runsc version release-20260824.0-93-gb3674d543f
spec: 1.2.1
```

## docker version (if using docker)

```shell
Client: Docker Engine - Community 26.1.3
Server: Docker Engine - Community 26.1.3
containerd: 1.6.32
```

## uname

`Linux 6.6.110`

## kubectl (if using Kubernetes)

## repo state (if built from source)

`release-20260824.0-93-gb3674d543f`

## runsc debug logs (if available)

Crashing goroutine only:

```shell
panic: unrecoverable overlayfs inconsistency: failed to create whiteout after unlinking upper layer file during UnlinkAt: no such file or directory

goroutine 42242 gp=0x664741b5860 m=20 mp=0x66473980808 [running]:
panic({0x131c6c0?, 0x6647513cab0?})
GOROOT/src/runtime/panic.go:879 +0x16f fp=0x66474635400 sp=0x66474635350 pc=0x48c94f
gvisor.dev/gvisor/pkg/sentry/fsimpl/overlay.(*filesystem).UnlinkAt(0x66473141200, {0x16afd78, 0x6647a6ccc88}, 0x664745fc488)
pkg/sentry/fsimpl/overlay/filesystem.go:1708 +0xa88 fp=0x66474635758 sp=0x66474635400 pc=0xaf8848
gvisor.dev/gvisor/pkg/sentry/vfs.(*VirtualFilesystem).UnlinkAt(0x66473303608, {0x16afd78, 0x6647a6ccc88}, 0x66479d15708?, 0x66473e75d00?)
pkg/sentry/vfs/vfs.go:755 +0x1ad fp=0x664746357f0 sp=0x66474635758 pc=0x8a8a6d
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.unlinkat(0x6647a6ccc88, 0xffffff9c, 0x66474635950?)
pkg/sentry/syscalls/linux/sys_file.go:1137 +0x1a8 fp=0x66474635960 sp=0x664746357f0 pc=0xd02e88
gvisor.dev/gvisor/pkg/sentry/syscalls/linux.Unlink(0x6647a6ccc88?, 0x1f?, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/syscalls/linux/sys_file.go:1124 +0x1d fp=0x66474635988 sp=0x66474635960 pc=0xd02c9d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).executeSyscall(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/kernel/task_syscall.go:143 +0x67d fp=0x66474635cc8 sp=0x66474635988 pc=0xb6913d
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallInvoke(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/kernel/task_syscall.go:323 +0x4b fp=0x66474635d20 sp=0x66474635cc8 pc=0xb6a2cb
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscallEnter(0x6647a6ccc88, 0x57, {{0x55c656cafef8}, {0x0}, {0x1}, {0x0}, {0x0}, {0x55c656a2acd4}})
pkg/sentry/kernel/task_syscall.go:283 +0x68 fp=0x66474635d70 sp=0x66474635d20 pc=0xb69fe8
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).doSyscall(0x66474316dc0?)
pkg/sentry/kernel/task_syscall.go:258 +0x2a5 fp=0x66474635e48 sp=0x66474635d70 pc=0xb69d65
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0x664743895c0?, 0x6647a6ccc88)
pkg/sentry/kernel/task_run.go:258 +0xf9c fp=0x66474635f30 sp=0x66474635e48 pc=0xb5e7fc
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0x6647a6ccc88, 0x2b6d)
pkg/sentry/kernel/task_run.go:89 +0x202 fp=0x66474635fc0 sp=0x66474635f30 pc=0xb5d362
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start.gowrap1()
pkg/sentry/kernel/task_start.go:566 +0x1b fp=0x66474635fe0 sp=0x66474635fc0 pc=0xb67c5b
runtime.goexit({})
src/runtime/asm_amd64.s:1771 +0x1 fp=0x66474635fe8 sp=0x66474635fe0 pc=0x495601
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start in goroutine 42119
pkg/sentry/kernel/task_start.go:566 +0xc5
```

Contributor guide

Open the contributing guide

Research direction

Start in pkg/sentry/fsimpl/overlay/filesystem.go at filesystem.UnlinkAt around line 1708, then inspect the overlay RenameAt path and its whiteout handling. Run the supplied nested-overlay reproduction to observe the race. Done means the concurrent unlink/rmdir case no longer panics and reports an appropriate filesystem error while the sandbox remains alive.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.