google / google/gvisor

shim: unbounded Init.mu hold across hung runsc kill; no escalation when the sentry stops answering

Open
#14,548 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

**Description**

When a sandbox becomes unresponsive, the shim (`containerd-shim-runsc-v1`) has no bound on how long it waits and no escalation. Two concrete gaps, both observed in production (#14405, #14408):

1. `Init.Kill()` / `killAllLocked()` holds `Init.mu` across the `runsc kill` subprocess with no timeout. If the sentry control server is wedged, that kill never returns, and everything behind the mutex queues up: in #14405 that was 45,646 goroutines (mostly kubelet/cadvisor `Stats` RPCs) and 610MB shim RSS after 5 days, with containerd and kubelet memory growing until node OOM.
2. There is no last-resort path. kubelet only talks CRI, containerd delegates to the shim, and the shim only talks to the sentry over URPC. When the sentry cannot answer, each layer's timeout bottoms out in asking the same wedged process again. The only unconditional remediation in the whole stack today is a human running `kill -9` on the sandbox process tree, which works instantly (PDEATHSIG/EXITKILL cascades and the pod gets reaped).

#14201 fixes the hang at its origin (a stuck systrap context now fail-stops inside the sentry), but the shim still has no protection against the next way a sentry finds to stop answering.

**Proposed behavior**

- Put a deadline on the `runsc kill` invocation in `killAllLocked()` (or stop holding `Init.mu` across it), so `Stats` and other RPCs cannot queue unboundedly behind one hung kill.
- After N consecutive kill attempts time out, escalate: SIGKILL the sandbox process tree directly from the shim. This is exactly what operators do by hand today; the shim is the lowest layer that knows all the pids and can do it safely.

If this direction sounds right il send a PR.

**Environment**

Seen with runsc release-20260427.0 and release-20260714.0 on EKS, containerd shim v2, systrap. Full forensics in #14408; goroutine dumps in #14405. Related upstream reports where users ended up doing the manual kill: containerd/containerd#8847, containerd/containerd#10891, containerd/containerd#11133.

Contributor guide

Open the contributing guide

Research direction

Start with Init.Kill() and killAllLocked(), then trace how the shim invokes runsc kill and tracks sandbox process IDs. Define how a kill deadline prevents work from queueing behind Init.mu and how repeated timeouts trigger direct process-tree termination. Done means hung sentry scenarios no longer block Stats indefinitely and the documented escalation path is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.