containerd / containerd/containerd

containerd 2.2.2 exits on SIGHUP instead of reloading — shim-log-after-reload fifo error

Open
#13,285 2 comments 0 reactions 0 assignees View on GitHub
area/runtime kind/feature
Dominant language
Go
Stars
21.3k
Forks
4.1k
Avg merge
2d 4h
Merged PRs (30d)
92

Description

### Description

Sending `SIGHUP` to a running `containerd` (to trigger config reload) causes containerd to exit instead of reloading. The process emits `Shutdown request received: "containerd exited: signal: hangup"` and supervisors (rke2, dockerd, etc.) tear down on the child exit.

Reproduces reliably in an RKE2 single-node cluster running the NVIDIA gpu-operator `nvidia-container-toolkit-ctr` DaemonSet, which issues `kill -HUP ` to apply its containerd drop-in. We confirmed the sender with the `signal_generate` tracepoint:

```
nvidia-toolkit-411570 signal_generate: sig=1 comm=containerd pid=369711
```

Sequence each time:
1. Toolkit container writes a containerd drop-in.
2. Toolkit calls `kill -HUP $CONTAINERD_PID`.
3. Containerd logs `"copy shim log after reload: file already closed"` and exits.
4. rke2's supervisor sees child exit → shuts down → systemd restarts → loop.

On this cluster the effect was a ~2-minute restart loop for the rke2-server service until the toolkit DaemonSet was removed from the cluster.

### Expected

`SIGHUP` should trigger a config reload per the documented behavior, not exit the daemon.

### Actual

containerd exits with `signal: hangup` immediately after receiving SIGHUP. The last log lines include:
```
level=error msg="copy shim log after reload: file already closed"
```
followed by the process exiting. Systemd/supervisor then logs:
```
level=error msg="Shutdown request received: \"containerd exited: signal: hangup\""
```

### Environment

- OS: Ubuntu 24.04.4 LTS
- Kernel: 6.14.0-33-generic (x86_64)
- Container runtime: containerd `v2.2.2-k3s1`, revision `281ac8b809cb6196b26be9720a904cf24738d436`, Go `go1.25.8 X:boringcrypto` (shipped by RKE2 v1.34.6+rke2r1 as `/var/lib/rancher/rke2/data/v1.34.6-rke2r1-5500c12bec0f/bin/containerd`)
- Supervisor: rke2 v1.34.6+rke2r1
- Trigger: nvidia-container-toolkit 1.19.0 daemonset (gpu-operator chart `gpu-operator-v26.3.0`)

### Reproduction (minimal)

On any host with a running containerd at a known PID, with root:
```bash
kill -HUP $(pgrep -x containerd)
# observe containerd exit via journalctl
```
On k3s/rke2 the exit manifests as the supervisor logging `Shutdown request received`.

### Workaround

Wrapping the containerd binary with a trivial SIGHUP-ignoring shell shim prevents the crash:
```bash
#!/bin/bash
trap '' HUP
exec /path/to/containerd.real "$@"
```
Cluster then stays stable, but the intended config reload does not happen. Root-cause fix should be in containerd's reload handler.

### Notes / related

- `"copy shim log after reload: file already closed"` matches what's seen when a shim has already cleaned up its log FIFOs but the reload path attempts to re-copy them.
- Related shim/fifo handling issues: containerd/containerd#6860, containerd/containerd#11033, docker/for-linux#1418 (different entry points, same family of fifo-lifecycle bugs).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.