e2b-dev / e2b-dev/runtime

orchestrator: NBD dispatch logs context.Canceled and closed-socket errors at ERROR level after normal VM exit

Open
#3,274 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
438
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

When a sandbox VM exits (normally or abnormally), the NBD dispatch loop in packages/orchestrator/pkg/sandbox/nbd/dispatch.go logs in-flight read/write failures at ERROR level, even when the errors are expected consequences of VM shutdown (context.Canceled, use of closed network connection). This produces false-alarm ERROR logs on every sandbox teardown.

Observed log lines

level=error msg="nbd backend read failed" error="context canceled" nbd_op=read ...
level=error msg="nbd error cmd read"      error="write unix @->@: use of closed network connection" ...
level=error msg="nbd backend write failed" error="context canceled" nbd_op=write ...
level=error msg="nbd error cmd write"     error="write unix @->@: write: broken pipe" ...

Root cause

In cmdRead() (line ~336) and cmdWrite() (line ~403), backend failures are unconditionally logged at ERROR regardless of whether the error is an expected shutdown signal:

// dispatch.go line 336
logger.L().Error(ctx, "nbd backend read failed",
    zap.Error(readErr), ...)

// dispatch.go line 358 — response write to closed socket
logger.L().Error(ctx, "nbd error cmd read",
    zap.Error(err), ...)

When the VM exits, the sandbox context is cancelled → context.Canceled propagates to all in-flight ReadAt/WriteAt calls and to socket writes on the now-closed NBD Unix socket. All of this is expected, but every in-flight request at the moment of VM exit emits an ERROR.

Expected behavior

  • context.Canceled / context.DeadlineExceeded backend errors → Debug or Warn
  • use of closed network connection / broken pipe response-write errors → Warn (the dispatch loop's fatal channel already handles the fatal case; the default: branch is only reached when the channel is full, meaning shutdown is already in progress)

Impact

  • Noisy ERROR logs on every sandbox teardown
  • Drowns out real storage/backend errors that should trigger alerts

Suggested fix

Differentiate by error type in the logging calls:

if errors.Is(readErr, context.Canceled) || errors.Is(readErr, context.DeadlineExceeded) {
    logger.L().Debug(ctx, "nbd backend read cancelled (VM exiting)", zap.Error(readErr), ...)
} else {
    logger.L().Error(ctx, "nbd backend read failed", zap.Error(readErr), ...)
}

Similarly for cmdWrite and the response-write error paths.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/orchestrator/pkg/sandbox/nbd/dispatch.go, reading cmdRead and cmdWrite plus the fatal-channel handling around response-write errors. Trace how context cancellation and closed-socket errors reach each logging call. Done means expected VM-shutdown errors no longer log at ERROR while genuine backend failures retain appropriate error-level logging.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.