kubernetes-sigs / kubernetes-sigs/node-feature-discovery
NFD worker does not handle SIGTERM gracefully
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 317
- Avg merge
- 21h 39m
- Merged PRs (30d)
- 5
Description
**What happened**:
nfd-worker exits with code 1 or 2 when receiving SIGTERM during pod termination (e.g., DaemonSet rolling updates). The process terminates abruptly without executing its graceful shutdown path. This triggers alerts in monitoring systems that track non-zero exit codes.
In a typical rollout, we observe:
- **Exit code 1** (most common): nfd-master is terminated first during the rollout. The worker's next gRPC call to master gets a connection error, `Run()` returns that error, and `main()`
hits `os.Exit(1)`.
- **Exit code 2** (less common): SIGTERM arrives while the process is idle (not mid-gRPC call). With no signal handler registered, the Go runtime terminates the process immediately.
Looking at the code in `cmd/nfd-worker/main.go`, the `main()` function creates an `NfdWorker` instance and calls `instance.Run()`, but never registers an OS signal handler to invoke `instance.Stop()` on SIGTERM/SIGINT:
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-worker/main.go#L38-L73
The worker internally has a clean shutdown path — `Stop()` closes a channel, and `Run()` receives it, logs "shutting down nfd-worker", and returns nil (exit 0):
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/pkg/nfd-worker/nfd-worker.go#L329-L346
However, since no signal handler bridges OS signals to `Stop()`, SIGTERM either kills the process directly (exit 2) or interrupts an in-flight gRPC call which propagates an error to `Run()` → `os.Exit(1)`.
**What you expected to happen**:
nfd-worker should catch SIGTERM/SIGINT via `signal.Notify`, call `Stop()`, and exit cleanly with code 0. This is the standard pattern for long-running Go processes in Kubernetes.
**How to reproduce it (as minimally and precisely as possible)**:
1. Deploy nfd-worker as a DaemonSet (e.g., via the NVIDIA gpu-operator helm chart which bundles NFD as a sub-chart)
2. Trigger a rolling update of the DaemonSet (e.g., update an annotation or image tag)
3. Observe that terminated pods report exit code 1 or 2 rather than 0
4. Alternatively: `kubectl exec` into a running nfd-worker pod and run `kill -TERM 1` — the process exits non-zero instead of shutting down gracefully
**Anything else we need to know?**:
The fix is straightforward — add signal handling in `cmd/nfd-worker/main.go`:
```go
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer cancel()
go func() {
<-ctx.Done()
instance.Stop()
}()
```
This same gap exists in all long-running NFD daemon entrypoints:
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-worker/main.go
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-master/main.go
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-gc/main.go
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-topology-updater/main.go
The worker's Run() already returns nil when Stop() is called, so wiring signals to Stop() is the only missing piece for a clean exit.
Environment:
- Kubernetes version: v1.32
- Cloud provider or hardware configuration: bare-metal (on-prem)
- OS: Ubuntu 24.04
- Kernel: N/A (not kernel-specific)
- Install tools: Helm (via NVIDIA gpu-operator chart, which includes NFD v0.15.4 as a sub-chart)
- Network plugin and version: N/A (not network-related)
- Others: Verified the signal handling gap exists in v0.15.4, v0.16.0, v0.18.3, and current master — the signal handling has never been added to the main entrypoints
Contributor guide
Research direction
Start with cmd/nfd-worker/main.go and compare its entrypoint with the shutdown path in pkg/nfd-worker/nfd-worker.go. Trace how Run() and Stop() interact, then inspect the daemon entrypoints in cmd/nfd-master/main.go, cmd/nfd-gc/main.go, and cmd/nfd-topology-updater/main.go. Done means SIGTERM and SIGINT trigger graceful shutdown and the daemons exit with code 0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100