NVIDIA / NVIDIA/nvcf

SIGTERM still panics worker-task, worker-llm-credentials, grpc-proxy, and vanity-gateway

Open
#1,449 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

What

#1319 fixed the SIGTERM-panic in worker-utils, but the same defect is still live in four other services. Each carries its own copy of the check, and each compares against the SIGINT wording only:

Service Site Check
worker-task service/service.go:73 strings.ToLower(err.Error()) != "received signal interrupt"
worker-llm-credentials service/service.go:49 strings.ToLower(err.Error()) != "received signal interrupt"
grpc-proxy main.go:37 err.Error() != "received signal interrupt"
vanity-gateway main.go:40 err.Error() != "received signal interrupt"

syscall.SIGTERM stringifies as terminated, so the error reads received signal terminated, does not match, and falls through to zap.S().Panic. SIGTERM is how Kubernetes asks a container to stop, so these checks excuse the signal that never arrives in production and panic on the one that always does. A routine pod termination is written into the termination log as a panic and reported as a crash.

worker-task/internal/worker/worker.go:260 gets the behavior right via a regexp covering both signals, but it is a third independent copy of the same logic.

Root cause

src/libraries/go/lib/pkg/nvkit/servers/grpc.go models a shutdown signal as a terminal error with no sentinel to test against:

return fmt.Errorf("received signal %s", sig)

Every consumer is forced to string-match, and there is nothing stopping the next one from matching on interrupt alone.

Fix

Two steps, split so that each auto-cut release tag is self-consistent:

  1. Add pkg/nvkit/shutdown to src/libraries/go/lib — a typed SignalError, an ErrSignal sentinel, and an IsSignalError predicate — and have servers return it. libraries/go/lib is not a released subproject, so this cuts no tags.
  2. Bump the lib pin in each consumer's go.mod and convert all five call sites to shutdown.IsSignalError. Every service here is a released subproject, so a fix: commit touching them auto-cuts a tag; doing the pin bump in the same change keeps those tags buildable for external consumers such as the GitLab go-nvcf-worker repo.

vanity-gateway takes its servers package from the separate github.com/NVIDIA/nvcf-go module, so it can never receive the sentinel. It relies on the predicate's message fallback, which is why that fallback is part of the design rather than a transitional shim.

Relates to

Follow-up to #1319, which called this out as deliberately out of scope.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/libraries/go/lib/pkg/nvkit/servers/grpc.go and the proposed shutdown package, then inspect the five call sites in the listed service.go, main.go, and worker.go files. Check the go.mod pins for each released consumer and the separate github.com/NVIDIA/nvcf-go dependency used by vanity-gateway. Done means SIGTERM and SIGINT avoid panic consistently and all affected services remain buildable with their updated lib pins.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.