Leftover containerd-shim-runsc-v1 processes after a failed restore make containerd hang forever on startup
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
### Description
## What happened
We run gVisor pods on Kubernetes. Each pod has two containers (a workload container and a
sidecar), and the pods are periodically checkpointed and restored using
`dev.gvisor.internal.restore.host-image-path`.
One of our pods failed to restore. In the containerd log we saw the restore fail with:
```
OCI runtime restore failed: starting container: starting sub-container [...]:
failed to load kernel: main MF start failed: failed to load metadata: header error: EOF
```
kubelet then retried the failing container a couple of times. Each retry logged this pattern:
```
CreateContainer within sandbox "517ef04c..." for name:"sandboxd" attempt:1 returns container id "baecc9ac..."
StartContainer for "baecc9ac..."
Failed to pipe stdout of container "baecc9ac..." error="read /proc/self/fd/456: file already closed"
Failed to pipe stderr of container "baecc9ac..." error="read /proc/self/fd/457: file already closed"
collecting metrics for baecc9ac... error="cannot stat a stopped container" <- repeats forever
failed to delete task error="context deadline exceeded" id=baecc9ac...
StartContainer for "baecc9ac..." failed error="... cannot start a stopped container"
```
Kubernetes then reported the pod as fully deleted — `StopPodSandbox` and `RemovePodSandbox` both
returned successfully, and `kubectl get pods` showed nothing.
**But two `containerd-shim-runsc-v1` processes for that pod were still running on the node**, and
they stayed running. `ctr -n k8s.io task ls` still listed them. Nothing we did through containerd
or crictl would remove them:
```
# ctr -n k8s.io task delete --force baecc9ac... -> hangs, never returns
# crictl rmp -> fails
```
Roughly 19 hours later, containerd restarted on that node for unrelated reasons — and never came
back up. It hangs during startup every single time:
```
# systemctl status containerd
Active: activating (start) since ...
# journalctl -u containerd
... cleaning leaked shim process" id=baecc9ac...
... (nothing further; systemd start-timeout kills it, then it repeats)
```
The node has been in this state for days: `NotReady`, with
```
container runtime is down, PLEG is not healthy: pleg was last seen active 3m30s ago
```
Every restart re-announces the same two shim IDs as
`cleaning leaked shim process` and then stops making progress.
The only thing that recovers the node is finding those shim PIDs and `kill -9`-ing them by hand,
then restarting containerd.
## Impact
- One container that fails to start can leave a shim process behind that we cannot remove through
any normal interface.
- The node keeps working fine afterwards, so nothing alerts. The problem only shows up the next
time containerd restarts, which for us was ~19 hours later — long enough that nothing obviously
connected the two events.
- When it does show up, containerd cannot start at all, so the node is down until someone logs in
and kills the leftover processes manually.
- We have hit this on three separate nodes.
## What we expected
That deleting a container which failed to start would either succeed or fail with an error, and
that containerd would be able to start even if a leftover shim cannot be cleaned up.
## Steps to reproduce
This reproduces deterministically for us. The key ingredient is a **corrupt restore image**: the
sandbox has to end up unusable while its process is still alive. (For what it's worth, `kill -9`
or `SIGSTOP` on the sandbox process does *not* reproduce it — a killed sandbox is noticed straight
away and kubelet just rebuilds the pod correctly.)
Tested on a Kubernetes node with `RuntimeClass: runsc`, `platform=systrap`, `overlay2=all:self`.
**1. Start a pod with two containers and `restartPolicy: Always`.** Both containers matter — the
problem only affects the non-root containers of a pod.
```yaml
kubectl apply -f -< 263d v1.36.2
worker-5 Ready 243d v1.36.2
worker-7 Ready 7d17h v1.36.2
master-1 Ready control-plane 266d v1.36.2
master-2 Ready control-plane 266d v1.36.2
master-3 Ready control-plane 266d v1.36.2
worker-1 Ready 266d v1.36.2
worker-2 Ready 266d v1.36.2
worker-3 Ready 266d v1.36.2
worker-8 Ready 93d v1.36.2
worker-9 Ready 93d v1.36.2
```
Happy to provide the full containerd journal, the `/proc` state of the leftover shims, or anything
else that would help.
Contributor guide
Research direction
The report names runsc restore, containerd-shim-runsc-v1 processes, and containerd's leaked-shim cleanup as the main entry points, but no repository files or tests. Start by reproducing the corrupt pages_meta.img restore sequence and tracing shim cleanup through containerd restart. Done means failed restores do not leave unremovable shims and containerd can start afterward.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, linux
- Domain
- devops, infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100