shim: unbounded Init.mu hold across hung runsc kill; no escalation when the sentry stops answering
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
**Description**
When a sandbox becomes unresponsive, the shim (`containerd-shim-runsc-v1`) has no bound on how long it waits and no escalation. Two concrete gaps, both observed in production (#14405, #14408):
1. `Init.Kill()` / `killAllLocked()` holds `Init.mu` across the `runsc kill` subprocess with no timeout. If the sentry control server is wedged, that kill never returns, and everything behind the mutex queues up: in #14405 that was 45,646 goroutines (mostly kubelet/cadvisor `Stats` RPCs) and 610MB shim RSS after 5 days, with containerd and kubelet memory growing until node OOM.
2. There is no last-resort path. kubelet only talks CRI, containerd delegates to the shim, and the shim only talks to the sentry over URPC. When the sentry cannot answer, each layer's timeout bottoms out in asking the same wedged process again. The only unconditional remediation in the whole stack today is a human running `kill -9` on the sandbox process tree, which works instantly (PDEATHSIG/EXITKILL cascades and the pod gets reaped).
#14201 fixes the hang at its origin (a stuck systrap context now fail-stops inside the sentry), but the shim still has no protection against the next way a sentry finds to stop answering.
**Proposed behavior**
- Put a deadline on the `runsc kill` invocation in `killAllLocked()` (or stop holding `Init.mu` across it), so `Stats` and other RPCs cannot queue unboundedly behind one hung kill.
- After N consecutive kill attempts time out, escalate: SIGKILL the sandbox process tree directly from the shim. This is exactly what operators do by hand today; the shim is the lowest layer that knows all the pids and can do it safely.
If this direction sounds right il send a PR.
**Environment**
Seen with runsc release-20260427.0 and release-20260714.0 on EKS, containerd shim v2, systrap. Full forensics in #14408; goroutine dumps in #14405. Related upstream reports where users ended up doing the manual kill: containerd/containerd#8847, containerd/containerd#10891, containerd/containerd#11133.
Contributor guide
Research direction
Start with Init.Kill() and killAllLocked(), then trace how the shim invokes runsc kill and tracks sandbox process IDs. Define how a kill deadline prevents work from queueing behind Init.mu and how repeated timeouts trigger direct process-tree termination. Done means hung sentry scenarios no longer block Stats indefinitely and the documented escalation path is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100