fix(orchestrator): terminate sandbox when UFFD handler fails
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Summary
The UFFD (userfaultfd) handler goroutine in uffd.go had a stale TODO comment claiming the sandbox is not killed when handle() fails. In fact, the sandbox is killed — but the failure was silent (no log entry), and the TODO left future readers believing the mechanism was unimplemented. There were also no unit tests covering handle() failure scenarios.
Root cause
The TODO comment was misleading:
// packages/orchestrator/pkg/sandbox/uffd/uffd.go
go func() {
// TODO: If the handle function fails, we should kill the sandbox ← STALE
handleErr := u.handle(ctx, sandboxId, fdExit)
...
u.exit.SetError(errors.Join(handleErr, closeErr, fdExitErr))
}()
The kill is already wired up. When u.exit.SetError() is called:
u.exit.Done()channel closes (viaErrorOnce→SetOnce.Done)- The sandbox lifecycle goroutine in
sandbox.go(lines 1164–1174) wakes immediately oncase <-fcUffd.Exit().Done() - It calls
sbx.Stop(ctx), which kills the Firecracker VM
// sandbox.go:1158–1174 — runs for the full sandbox lifetime
go func() {
// Wait for either uffd or fc process to exit.
select {
case <-fcUffd.Exit().Done(): // fires on UFFD handle() failure
case <-fcHandle.Exit.Done():
}
err := sbx.Stop(ctx) // actively kills the sandbox
...
}()
Real problems
- Stale TODO — the comment stated the kill was not implemented, causing confusion and risking someone "fixing" the already-correct behaviour in an incorrect way.
- No logging — when
handle()fails, no error is emitted. Failures are invisible in production logs. - No test coverage — no tests exercised the exit-error propagation path (
handle()fail →u.exit.SetError()→Exit().Done()closes), leaving the behaviour unverified.
Impact
- Production UFFD failures are invisible (no log line, no alert surface).
- Sandbox is terminated automatically, so there is no zombie-sandbox risk.
- The stale TODO creates a correctness trap: a reader assuming the TODO describes a real gap may add redundant or conflicting kill logic.
Proposed fix
- Remove the stale TODO.
- Add an error-level log line when
handle()fails, so failures are observable. - Add unit tests covering: exit error propagation,
readyChclosure,handlererror state, initial state, and socket creation onStart(). - Add a comment to the
sandbox.golifecycle goroutine explaining that UFFD failure flows throughfcUffd.Exit().Done()→sbx.Stop(), making the existing mechanism explicit.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/orchestrator/pkg/sandbox/uffd/uffd.go, reading Start(), handle(), and the exit and ready-state paths; then trace the lifecycle select in sandbox.go around lines 1158–1174. Run the UFFD package tests and verify that handle() failures are logged, exit propagation and channel/state behavior are covered, socket creation on Start() is tested, and the lifecycle comment reflects the existing stop path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100