orchestrator: NBD dispatch logs context.Canceled and closed-socket errors at ERROR level after normal VM exit
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
When a sandbox VM exits (normally or abnormally), the NBD dispatch loop in packages/orchestrator/pkg/sandbox/nbd/dispatch.go logs in-flight read/write failures at ERROR level, even when the errors are expected consequences of VM shutdown (context.Canceled, use of closed network connection). This produces false-alarm ERROR logs on every sandbox teardown.
Observed log lines
level=error msg="nbd backend read failed" error="context canceled" nbd_op=read ...
level=error msg="nbd error cmd read" error="write unix @->@: use of closed network connection" ...
level=error msg="nbd backend write failed" error="context canceled" nbd_op=write ...
level=error msg="nbd error cmd write" error="write unix @->@: write: broken pipe" ...
Root cause
In cmdRead() (line ~336) and cmdWrite() (line ~403), backend failures are unconditionally logged at ERROR regardless of whether the error is an expected shutdown signal:
// dispatch.go line 336
logger.L().Error(ctx, "nbd backend read failed",
zap.Error(readErr), ...)
// dispatch.go line 358 — response write to closed socket
logger.L().Error(ctx, "nbd error cmd read",
zap.Error(err), ...)
When the VM exits, the sandbox context is cancelled → context.Canceled propagates to all in-flight ReadAt/WriteAt calls and to socket writes on the now-closed NBD Unix socket. All of this is expected, but every in-flight request at the moment of VM exit emits an ERROR.
Expected behavior
context.Canceled/context.DeadlineExceededbackend errors → Debug or Warnuse of closed network connection/broken piperesponse-write errors → Warn (the dispatch loop'sfatalchannel already handles the fatal case; thedefault:branch is only reached when the channel is full, meaning shutdown is already in progress)
Impact
- Noisy ERROR logs on every sandbox teardown
- Drowns out real storage/backend errors that should trigger alerts
Suggested fix
Differentiate by error type in the logging calls:
if errors.Is(readErr, context.Canceled) || errors.Is(readErr, context.DeadlineExceeded) {
logger.L().Debug(ctx, "nbd backend read cancelled (VM exiting)", zap.Error(readErr), ...)
} else {
logger.L().Error(ctx, "nbd backend read failed", zap.Error(readErr), ...)
}
Similarly for cmdWrite and the response-write error paths.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/orchestrator/pkg/sandbox/nbd/dispatch.go, reading cmdRead and cmdWrite plus the fatal-channel handling around response-write errors. Trace how context cancellation and closed-socket errors reach each logging call. Done means expected VM-shutdown errors no longer log at ERROR while genuine backend failures retain appropriate error-level logging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100