orchestrator: NBD dispatch response-write errors (EPIPE) logged at ERROR on normal VM exit
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Problem
After a sandbox VM exits, the NBD client-side socket closes. Any in-flight writeResponse call to that socket (sending a read/write reply) receives EPIPE. These errors are currently logged at ERROR in three sites inside packages/orchestrator/pkg/sandbox/nbd/dispatch.go:
- Line ~362:
logger.L().Error(ctx, "nbd error cmd read", ...) - Line ~422:
logger.L().Error(ctx, "nbd error cmd write", ...) - Line ~500:
logger.L().Error(ctx, "nbd error cmd write-zeroes", ...)
All three sites fire when writeResponse fails and d.fatal is already full (i.e., a fatal error was already reported). On a normally-exiting VM this sequence is:
- VM quits → NBD socket closed → dispatch loop reads EOF/EPIPE from the socket → sends to
d.fatal - Any concurrent in-flight read/write goroutine tries to call
writeResponse→ EPIPE → tries to send tod.fatal(already full) → falls into thedefaultbranch → logs ERROR
This is expected behaviour during normal VM shutdown and should be logged at WARN or lower, not ERROR.
Relation to #3274
Issue #3274 tracks context.Canceled logged at ERROR in the backend-read path ("nbd backend read failed"). This issue is about the response-write path ("nbd error cmd read/write/write-zeroes") which fires on EPIPE from the NBD Unix socket.
Fix
logFn := logger.L().Error
if errors.Is(err, syscall.EPIPE) || errors.Is(err, io.ErrClosedPipe) {
logFn = logger.L().Warn
}
logFn(ctx, "nbd error cmd read", ...)
Apply the same pattern to all three sites. This change exists in the local branch fix/nbd-broken-pipe-log-level (commits 618d4d1, 08d6a58) but has not been merged to main.
Impact
Log noise — ERROR-level alerts fire on every normal sandbox exit that has in-flight NBD operations at exit time.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read packages/orchestrator/pkg/sandbox/nbd/dispatch.go and inspect the three response-write error sites around lines 362, 422, and 500. Confirm how EPIPE and io.ErrClosedPipe are handled, then verify that normal VM shutdown no longer produces ERROR logs for these paths using the package's existing tests or checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100