e2b-dev / e2b-dev/runtime

fix(orchestrator): memfile.header not written in Local storage mode — templates break after cache eviction

Open
#3,248 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.6k
Forks
438
PR merge metrics
No merged PRs in 30d

Description

Problem

When using STORAGE_PROVIDER=Local, newly built templates work for a few minutes then permanently fail with:

object does not exist
failed to get snapfile: NEW STORAGE failed to write to file: object does not exist
[FailedPrecondition] sandbox files for '<id>' not found

Root cause

In build_upload_v3.go / build_upload_v4.go, the memfile header and body are uploaded in separate goroutines:

eg.Go(func() error {
    h, err := u.snap.MemorySnapshot.DiffHeader.WaitWithContext(egCtx)
    if err != nil { return ... }
    if h == nil { return nil }   // ← returns without writing header
    return storeHeaderWithMetrics(...)
})

In the Local storage path, DiffHeader.WaitWithContext resolves to nil, so the header goroutine exits cleanly without writing memfile.header. The body goroutine writes memfile successfully. The build is then marked uploaded even though the snapshot is incomplete.

The orchestrator's in-memory snapshot cache serves sandboxes correctly for a few minutes. Once the cache entry is evicted, every subsequent Sandbox.create hits local storage, finds no memfile.header, and fails with FailedPrecondition.

Expected behaviour

memfile.header must be written to local storage on every successful template build that produces a memory snapshot.

Observed on-disk state

  • base (pre-bundled): memfile ✓, memfile.header ✓, rootfs ✓, snapfile ✓
  • user-built template: memfile ✓, memfile.header missing, rootfs ✓, snapfile ✓

Reported by

#3226

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with build_upload_v3.go and build_upload_v4.go, then trace the DiffHeader.WaitWithContext handling in the Local storage path. Confirm how successful memory snapshots write both memfile and memfile.header, and validate the on-disk state after the in-memory snapshot cache is evicted. Done means locally built templates continue to support Sandbox.create after cache eviction.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.