containerd / containerd/containerd
containerd 2.2.2 exits on SIGHUP instead of reloading — shim-log-after-reload fifo error
- Dominant language
- Go
- Stars
- 21.3k
- Forks
- 4.1k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 92
Description
### Description
Sending `SIGHUP` to a running `containerd` (to trigger config reload) causes containerd to exit instead of reloading. The process emits `Shutdown request received: "containerd exited: signal: hangup"` and supervisors (rke2, dockerd, etc.) tear down on the child exit.
Reproduces reliably in an RKE2 single-node cluster running the NVIDIA gpu-operator `nvidia-container-toolkit-ctr` DaemonSet, which issues `kill -HUP ` to apply its containerd drop-in. We confirmed the sender with the `signal_generate` tracepoint:
```
nvidia-toolkit-411570 signal_generate: sig=1 comm=containerd pid=369711
```
Sequence each time:
1. Toolkit container writes a containerd drop-in.
2. Toolkit calls `kill -HUP $CONTAINERD_PID`.
3. Containerd logs `"copy shim log after reload: file already closed"` and exits.
4. rke2's supervisor sees child exit → shuts down → systemd restarts → loop.
On this cluster the effect was a ~2-minute restart loop for the rke2-server service until the toolkit DaemonSet was removed from the cluster.
### Expected
`SIGHUP` should trigger a config reload per the documented behavior, not exit the daemon.
### Actual
containerd exits with `signal: hangup` immediately after receiving SIGHUP. The last log lines include:
```
level=error msg="copy shim log after reload: file already closed"
```
followed by the process exiting. Systemd/supervisor then logs:
```
level=error msg="Shutdown request received: \"containerd exited: signal: hangup\""
```
### Environment
- OS: Ubuntu 24.04.4 LTS
- Kernel: 6.14.0-33-generic (x86_64)
- Container runtime: containerd `v2.2.2-k3s1`, revision `281ac8b809cb6196b26be9720a904cf24738d436`, Go `go1.25.8 X:boringcrypto` (shipped by RKE2 v1.34.6+rke2r1 as `/var/lib/rancher/rke2/data/v1.34.6-rke2r1-5500c12bec0f/bin/containerd`)
- Supervisor: rke2 v1.34.6+rke2r1
- Trigger: nvidia-container-toolkit 1.19.0 daemonset (gpu-operator chart `gpu-operator-v26.3.0`)
### Reproduction (minimal)
On any host with a running containerd at a known PID, with root:
```bash
kill -HUP $(pgrep -x containerd)
# observe containerd exit via journalctl
```
On k3s/rke2 the exit manifests as the supervisor logging `Shutdown request received`.
### Workaround
Wrapping the containerd binary with a trivial SIGHUP-ignoring shell shim prevents the crash:
```bash
#!/bin/bash
trap '' HUP
exec /path/to/containerd.real "$@"
```
Cluster then stays stable, but the intended config reload does not happen. Root-cause fix should be in containerd's reload handler.
### Notes / related
- `"copy shim log after reload: file already closed"` matches what's seen when a shim has already cleaned up its log FIFOs but the reload path attempts to re-copy them.
- Related shim/fifo handling issues: containerd/containerd#6860, containerd/containerd#11033, docker/for-linux#1418 (different entry points, same family of fifo-lifecycle bugs).
Contributor guide
Assessment
This issue has not been assessed yet.