kubernetes-sigs / kubernetes-sigs/node-feature-discovery

NFD worker does not handle SIGTERM gracefully

Open
#2,546 0 comments 1 reaction 0 assignees View on GitHub
kind/bug
Dominant language
Go
Stars
1.1k
Forks
317
Avg merge
21h 39m
Merged PRs (30d)
5

Description

**What happened**:

nfd-worker exits with code 1 or 2 when receiving SIGTERM during pod termination (e.g., DaemonSet rolling updates). The process terminates abruptly without executing its graceful shutdown path. This triggers alerts in monitoring systems that track non-zero exit codes.

In a typical rollout, we observe:
- **Exit code 1** (most common): nfd-master is terminated first during the rollout. The worker's next gRPC call to master gets a connection error, `Run()` returns that error, and `main()`
hits `os.Exit(1)`.
- **Exit code 2** (less common): SIGTERM arrives while the process is idle (not mid-gRPC call). With no signal handler registered, the Go runtime terminates the process immediately.

Looking at the code in `cmd/nfd-worker/main.go`, the `main()` function creates an `NfdWorker` instance and calls `instance.Run()`, but never registers an OS signal handler to invoke `instance.Stop()` on SIGTERM/SIGINT:

https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-worker/main.go#L38-L73

The worker internally has a clean shutdown path — `Stop()` closes a channel, and `Run()` receives it, logs "shutting down nfd-worker", and returns nil (exit 0):

https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/pkg/nfd-worker/nfd-worker.go#L329-L346

However, since no signal handler bridges OS signals to `Stop()`, SIGTERM either kills the process directly (exit 2) or interrupts an in-flight gRPC call which propagates an error to `Run()` → `os.Exit(1)`.

**What you expected to happen**:

nfd-worker should catch SIGTERM/SIGINT via `signal.Notify`, call `Stop()`, and exit cleanly with code 0. This is the standard pattern for long-running Go processes in Kubernetes.

**How to reproduce it (as minimally and precisely as possible)**:

1. Deploy nfd-worker as a DaemonSet (e.g., via the NVIDIA gpu-operator helm chart which bundles NFD as a sub-chart)
2. Trigger a rolling update of the DaemonSet (e.g., update an annotation or image tag)
3. Observe that terminated pods report exit code 1 or 2 rather than 0
4. Alternatively: `kubectl exec` into a running nfd-worker pod and run `kill -TERM 1` — the process exits non-zero instead of shutting down gracefully

**Anything else we need to know?**:

The fix is straightforward — add signal handling in `cmd/nfd-worker/main.go`:

```go
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer cancel()

go func() {
<-ctx.Done()
instance.Stop()
}()
```

This same gap exists in all long-running NFD daemon entrypoints:

https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-worker/main.go
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-master/main.go
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-gc/main.go
https://github.com/kubernetes-sigs/node-feature-discovery/blob/v0.18.3/cmd/nfd-topology-updater/main.go

The worker's Run() already returns nil when Stop() is called, so wiring signals to Stop() is the only missing piece for a clean exit.

Environment:
- Kubernetes version: v1.32
- Cloud provider or hardware configuration: bare-metal (on-prem)
- OS: Ubuntu 24.04
- Kernel: N/A (not kernel-specific)
- Install tools: Helm (via NVIDIA gpu-operator chart, which includes NFD v0.15.4 as a sub-chart)
- Network plugin and version: N/A (not network-related)
- Others: Verified the signal handling gap exists in v0.15.4, v0.16.0, v0.18.3, and current master — the signal handling has never been added to the main entrypoints

Contributor guide

Open the contributing guide

Research direction

Start with cmd/nfd-worker/main.go and compare its entrypoint with the shutdown path in pkg/nfd-worker/nfd-worker.go. Trace how Run() and Stop() interact, then inspect the daemon entrypoints in cmd/nfd-master/main.go, cmd/nfd-gc/main.go, and cmd/nfd-topology-updater/main.go. Done means SIGTERM and SIGINT trigger graceful shutdown and the daemons exit with code 0.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.