SIGTERM still panics worker-task, worker-llm-credentials, grpc-proxy, and vanity-gateway
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
What
#1319 fixed the SIGTERM-panic in worker-utils, but the same defect is still live in four other services. Each carries its own copy of the check, and each compares against the SIGINT wording only:
| Service | Site | Check |
|---|---|---|
| worker-task | service/service.go:73 |
strings.ToLower(err.Error()) != "received signal interrupt" |
| worker-llm-credentials | service/service.go:49 |
strings.ToLower(err.Error()) != "received signal interrupt" |
| grpc-proxy | main.go:37 |
err.Error() != "received signal interrupt" |
| vanity-gateway | main.go:40 |
err.Error() != "received signal interrupt" |
syscall.SIGTERM stringifies as terminated, so the error reads received signal terminated, does not match, and falls through to zap.S().Panic. SIGTERM is how Kubernetes asks a container to stop, so these checks excuse the signal that never arrives in production and panic on the one that always does. A routine pod termination is written into the termination log as a panic and reported as a crash.
worker-task/internal/worker/worker.go:260 gets the behavior right via a regexp covering both signals, but it is a third independent copy of the same logic.
Root cause
src/libraries/go/lib/pkg/nvkit/servers/grpc.go models a shutdown signal as a terminal error with no sentinel to test against:
return fmt.Errorf("received signal %s", sig)
Every consumer is forced to string-match, and there is nothing stopping the next one from matching on interrupt alone.
Fix
Two steps, split so that each auto-cut release tag is self-consistent:
- Add
pkg/nvkit/shutdowntosrc/libraries/go/lib— a typedSignalError, anErrSignalsentinel, and anIsSignalErrorpredicate — and haveserversreturn it.libraries/go/libis not a released subproject, so this cuts no tags. - Bump the
libpin in each consumer'sgo.modand convert all five call sites toshutdown.IsSignalError. Every service here is a released subproject, so afix:commit touching them auto-cuts a tag; doing the pin bump in the same change keeps those tags buildable for external consumers such as the GitLabgo-nvcf-workerrepo.
vanity-gateway takes its servers package from the separate github.com/NVIDIA/nvcf-go module, so it can never receive the sentinel. It relies on the predicate's message fallback, which is why that fallback is part of the design rather than a transitional shim.
Relates to
Follow-up to #1319, which called this out as deliberately out of scope.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/libraries/go/lib/pkg/nvkit/servers/grpc.go and the proposed shutdown package, then inspect the five call sites in the listed service.go, main.go, and worker.go files. Check the go.mod pins for each released consumer and the separate github.com/NVIDIA/nvcf-go dependency used by vanity-gateway. Done means SIGTERM and SIGINT avoid panic consistently and all affected services remain buildable with their updated lib pins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100