e2b-dev / e2b-dev/runtime

fix(orchestrator): terminate sandbox when UFFD handler fails

Open
#3,131 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
438
PR merge metrics
No merged PRs in 30d

Description

Summary

The UFFD (userfaultfd) handler goroutine in uffd.go had a stale TODO comment claiming the sandbox is not killed when handle() fails. In fact, the sandbox is killed — but the failure was silent (no log entry), and the TODO left future readers believing the mechanism was unimplemented. There were also no unit tests covering handle() failure scenarios.

Root cause

The TODO comment was misleading:

// packages/orchestrator/pkg/sandbox/uffd/uffd.go
go func() {
    // TODO: If the handle function fails, we should kill the sandbox  ← STALE
    handleErr := u.handle(ctx, sandboxId, fdExit)
    ...
    u.exit.SetError(errors.Join(handleErr, closeErr, fdExitErr))
}()

The kill is already wired up. When u.exit.SetError() is called:

  1. u.exit.Done() channel closes (via ErrorOnceSetOnce.Done)
  2. The sandbox lifecycle goroutine in sandbox.go (lines 1164–1174) wakes immediately on case <-fcUffd.Exit().Done()
  3. It calls sbx.Stop(ctx), which kills the Firecracker VM
// sandbox.go:1158–1174 — runs for the full sandbox lifetime
go func() {
    // Wait for either uffd or fc process to exit.
    select {
    case <-fcUffd.Exit().Done():   // fires on UFFD handle() failure
    case <-fcHandle.Exit.Done():
    }
    err := sbx.Stop(ctx)           // actively kills the sandbox
    ...
}()

Real problems

  1. Stale TODO — the comment stated the kill was not implemented, causing confusion and risking someone "fixing" the already-correct behaviour in an incorrect way.
  2. No logging — when handle() fails, no error is emitted. Failures are invisible in production logs.
  3. No test coverage — no tests exercised the exit-error propagation path (handle() fail → u.exit.SetError()Exit().Done() closes), leaving the behaviour unverified.

Impact

  • Production UFFD failures are invisible (no log line, no alert surface).
  • Sandbox is terminated automatically, so there is no zombie-sandbox risk.
  • The stale TODO creates a correctness trap: a reader assuming the TODO describes a real gap may add redundant or conflicting kill logic.

Proposed fix

  1. Remove the stale TODO.
  2. Add an error-level log line when handle() fails, so failures are observable.
  3. Add unit tests covering: exit error propagation, readyCh closure, handler error state, initial state, and socket creation on Start().
  4. Add a comment to the sandbox.go lifecycle goroutine explaining that UFFD failure flows through fcUffd.Exit().Done()sbx.Stop(), making the existing mechanism explicit.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/orchestrator/pkg/sandbox/uffd/uffd.go, reading Start(), handle(), and the exit and ready-state paths; then trace the lifecycle select in sandbox.go around lines 1158–1174. Run the UFFD package tests and verify that handle() failures are logged, exit propagation and channel/state behavior are covered, socket creation on Start() is tested, and the lifecycle comment reflects the existing stop path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.