overlay: sandbox can never exit after a self-deadlock on `renameMu` when an inotify Notify under `doCreateAt` removes an expired watch

**Description**

A container running on the overlay filesystem (`--overlay2`, default `root:self`) stops responding to SIGKILL. `runsc state` reports `stopped` with pid -1 while `runsc ps` still lists the workload processes, `runsc kill` returns success and nothing happens, and the shim's Wait never returns, so containerd and the kubelet retry the stop forever. The sentry itself is healthy: it answers `runsc debug --stacks`, and the dump shows one goroutine that has requested `renameMu` for write while the same goroutine already holds it for read. Every later `overlay.(*dentry).DecRef`, including the ones on the task-exit path, queues behind it, so the exit can never complete.

We have 43 sandboxes in this state, the oldest blocked for over 44 hours at the time of the dump.

**Stack (condensed, innermost first, `runsc debug --stacks`)**

```
goroutine 257515 [sync.RWMutex.Lock, 2656 minutes]:
overlay.(*renameRWMutex).Lock
overlay.(*dentry).OnZeroWatches <- wants renameMu for write
vfs.(*Inotify).RmWatch
vfs.(*Watches).cleanupExpiredWatches
vfs.(*Watches).Notify
overlay.(*filesystem).doCreateAt <- holds renameMu for read (deferred unlock)
overlay.(*filesystem).MkdirAt
... mkdir(2)

2 x goroutine [sync.RWMutex.Lock, 2628 minutes]:
overlay.(*renameRWMutex).Lock
overlay.(*dentry).DecRef
vfs.(*FileDescription).DecRef
mm.(*MemoryManager).DecUsers
kernel.(*runExitMain).execute <- the SIGKILL exit path, blocked behind the first goroutine

goroutine 1: kernel.(*Kernel).WaitExited (never returns)
goroutine 161: kernel.(*ThreadGroup).WaitExited (the shim's Wait)
```

Full dump available on request.

**Analysis**

Three pieces of code, unchanged between release-20260622.0, release-20260831.0 and master at d94a98aead1b:

- `pkg/sentry/fsimpl/overlay/filesystem.go`, `doCreateAt`: takes `fs.renameMu.RLock()` with `defer fs.renameMuRUnlockAndCheckDrop(ctx, &ds)` and calls `parent.watches.Notify(...)` before returning, so the notification runs under the read lock.
- `pkg/sentry/vfs/inotify.go`: when `Notify` finds watches whose inotify instance has been released, `cleanupExpiredWatches` calls `Inotify.RmWatch`, which calls `w.target.OnZeroWatches(ctx)` synchronously on the notifying goroutine once the target has no watches left.
- `pkg/sentry/fsimpl/overlay/overlay.go`, `(*dentry).OnZeroWatches`: `if d.refs.Load() == 0 { d.fs.renameMu.Lock(); d.checkDropLocked(ctx); d.fs.renameMu.Unlock() }`.

Walked directories in the overlay normally sit at zero references (they are kept alive by the parent's child map under `renameMu`, which is why the drop-list exists), so the condition holds for the parent of a create, and the write lock is requested by the goroutine that holds the read lock. Go's RWMutex never grants that.

The `DentryImpl.OnZeroWatches` contract only says no inotify locks may be held by the caller; it does not say anything about filesystem locks, and the overlay calls it from a path holding its own. The gofer filesystem's `OnZeroWatches` has the same shape through `checkCachingLocked` and its create paths also notify under the read lock, so the same hazard probably exists there; we have not observed it.

**Sequence that triggers it**

1. A process adds an inotify watch with `IN_ONESHOT` on a leaf directory of the overlay (a directory with no cached children, so its dentry sits at zero references) and keeps the inotify fd open. `IN_ONESHOT` is the only path that marks a watch expired and leaves its removal to `Watches.Notify`; releasing an inotify instance removes its watches synchronously with no filesystem lock held, so a watcher exiting is not a trigger.
2. Another process produces the first event on that directory: `mkdir` inside it (`doCreateAt`, `renameMu.RLock`) or a rename of it (`RenameAt`, `renameMu.Lock`, through `InotifyRename` / `IN_MOVE_SELF`). `Watch.Notify` fires the one-shot watch and marks it expired; the same `Notify` call runs `cleanupExpiredWatches -> RmWatch -> OnZeroWatches`, which requests `renameMu.Lock` on the goroutine that already holds `renameMu`, and the sandbox is wedged.
3. `open(O_CREAT)` in the same directory does not trigger it: `getChildLocked` takes a parent reference before the notification, so `OnZeroWatches` sees `refs != 0` and skips the lock.

Reproducer inside a container (alpine 3.21 rootfs with python3; the path must be on the overlay-backed rootfs, not under `/tmp`, which runsc backs with a tmpfs whose `OnZeroWatches` is a no-op):

```sh
mkdir -p /work/w/sub
python3 - <<'EOF' &
import ctypes, time
libc = ctypes.CDLL(None, use_errno=True)
fd = libc.inotify_init()
IN_ALL_EVENTS, IN_ONESHOT = 0x00000fff, 0x80000000
for p in (b"/work/w", b"/work/w/sub"):
libc.inotify_add_watch(fd, p, IN_ALL_EVENTS | IN_ONESHOT)
time.sleep(3600) # keep the fd open, never read it
EOF
sleep 1
mkdir /work/w/sub/new # hangs forever
# or: mv /work/w/sub /work/w/sub2
```

Results with runsc built from the `go` branch at 1db01ca (release-20260831.0-42), platform systrap, `runsc run` with the default `--overlay2=root:self` (`root:memory` behaves the same):

| binary | `mkdir` in the watched leaf dir | `mv` of the watched leaf dir | `open(O_CREAT)` in it |
|---|---|---|---|
| 1db01ca | hangs | hangs | completes |
| 1db01ca + #14684 | completes in 10-20 ms, container exits 0 | same | completes |

On the hung sandbox: `runsc kill -all KILL` never returns (blocked in `Kernel.Pause`, because the task inside `mkdirat` never reaches a stop point); `runsc kill KILL` returns 0, `runsc state` then reports `stopped` / `pid -1` while `runsc ps` still lists the processes, and the init task's exit path is blocked in `runExitMain -> FDTable.DecRef -> overlay.(*dentry).DecRef -> renameMu.Lock`. Only `runsc delete -force` (SIGKILL of the sandbox process) ends it. The stacks match the production dump above frame for frame; the sentry watchdog logs `Sentry detected 1 stuck task(s)` every 45 s and, with the default `--watchdog-action=log`, does nothing else.

**Possible fixes**

- Issue the inotify notification after `renameMuRUnlockAndCheckDrop`, holding a reference on the parent across the call. Every `Notify` issued while `renameMu` is held is exposed, not only the one in `doCreateAt`.
- Or make the overlay's `OnZeroWatches` defer the drop (queue the dentry for the next `renameMuRUnlockAndCheckDrop`) instead of taking the write lock inline.
- Or, at the vfs level, run `OnZeroWatches` from the expired-watch cleanup on its own goroutine and leave the `inotify_rm_watch` path synchronous: #14684, verified against the reproducer above.

**Environment**

- runsc release-20260622.0, platform systrap, `--overlay2` default; the relevant code is identical in release-20260831.0 and master d94a98aead1b.
- containerd 2.2.5 with containerd-shim-runsc-v1, Kubernetes 1.35, Linux 6.12 x86_64.
- Workload: Racket builds and tests. Racket's runtime adds its inotify watches with `IN_ONESHOT` (`racket/src/rktio/rktio_fs_change.c`) on the directories it watches (`racket/src/expander/eval/collection.rkt`), and the build creates directories under them.

Start with pkg/sentry/fsimpl/overlay/filesystem.go, pkg/sentry/vfs/inotify.go, and pkg/sentry/fsimpl/overlay/overlay.go, then run the provided inotify reproducer on the overlay-backed rootfs. Trace doCreateAt, Notify cleanup, and OnZeroWatches to understand the lock interaction. Done means mkdir and rename no longer wedge the sandbox, and the container exits normally after the reproducer.

Written by the indexing model from the issue text.