e2b-dev / e2b-dev/runtime

orchestrator: NBD dispatch response-write errors (EPIPE) logged at ERROR on normal VM exit

Open
#3,348 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
438
PR merge metrics
No merged PRs in 30d

Description

Problem

After a sandbox VM exits, the NBD client-side socket closes. Any in-flight writeResponse call to that socket (sending a read/write reply) receives EPIPE. These errors are currently logged at ERROR in three sites inside packages/orchestrator/pkg/sandbox/nbd/dispatch.go:

  • Line ~362: logger.L().Error(ctx, "nbd error cmd read", ...)
  • Line ~422: logger.L().Error(ctx, "nbd error cmd write", ...)
  • Line ~500: logger.L().Error(ctx, "nbd error cmd write-zeroes", ...)

All three sites fire when writeResponse fails and d.fatal is already full (i.e., a fatal error was already reported). On a normally-exiting VM this sequence is:

  1. VM quits → NBD socket closed → dispatch loop reads EOF/EPIPE from the socket → sends to d.fatal
  2. Any concurrent in-flight read/write goroutine tries to call writeResponse → EPIPE → tries to send to d.fatal (already full) → falls into the default branch → logs ERROR

This is expected behaviour during normal VM shutdown and should be logged at WARN or lower, not ERROR.

Relation to #3274

Issue #3274 tracks context.Canceled logged at ERROR in the backend-read path ("nbd backend read failed"). This issue is about the response-write path ("nbd error cmd read/write/write-zeroes") which fires on EPIPE from the NBD Unix socket.

Fix

logFn := logger.L().Error
if errors.Is(err, syscall.EPIPE) || errors.Is(err, io.ErrClosedPipe) {
    logFn = logger.L().Warn
}
logFn(ctx, "nbd error cmd read", ...)

Apply the same pattern to all three sites. This change exists in the local branch fix/nbd-broken-pipe-log-level (commits 618d4d1, 08d6a58) but has not been merged to main.

Impact

Log noise — ERROR-level alerts fire on every normal sandbox exit that has in-flight NBD operations at exit time.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read packages/orchestrator/pkg/sandbox/nbd/dispatch.go and inspect the three response-write error sites around lines 362, 422, and 500. Confirm how EPIPE and io.ErrClosedPipe are handled, then verify that normal VM shutdown no longer produces ERROR logs for these paths using the package's existing tests or checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.