fix(orchestrator): memfile.header not written in Local storage mode — templates break after cache eviction
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 438
- PR merge metrics
- No merged PRs in 30d
Description
Problem
When using STORAGE_PROVIDER=Local, newly built templates work for a few minutes then permanently fail with:
object does not exist
failed to get snapfile: NEW STORAGE failed to write to file: object does not exist
[FailedPrecondition] sandbox files for '<id>' not found
Root cause
In build_upload_v3.go / build_upload_v4.go, the memfile header and body are uploaded in separate goroutines:
eg.Go(func() error {
h, err := u.snap.MemorySnapshot.DiffHeader.WaitWithContext(egCtx)
if err != nil { return ... }
if h == nil { return nil } // ← returns without writing header
return storeHeaderWithMetrics(...)
})
In the Local storage path, DiffHeader.WaitWithContext resolves to nil, so the header goroutine exits cleanly without writing memfile.header. The body goroutine writes memfile successfully. The build is then marked uploaded even though the snapshot is incomplete.
The orchestrator's in-memory snapshot cache serves sandboxes correctly for a few minutes. Once the cache entry is evicted, every subsequent Sandbox.create hits local storage, finds no memfile.header, and fails with FailedPrecondition.
Expected behaviour
memfile.header must be written to local storage on every successful template build that produces a memory snapshot.
Observed on-disk state
- base (pre-bundled): memfile ✓, memfile.header ✓, rootfs ✓, snapfile ✓
- user-built template: memfile ✓, memfile.header missing, rootfs ✓, snapfile ✓
Reported by
#3226
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with build_upload_v3.go and build_upload_v4.go, then trace the DiffHeader.WaitWithContext handling in the Local storage path. Confirm how successful memory snapshots write both memfile and memfile.header, and validate the on-disk state after the in-memory snapshot cache is evicted. Done means locally built templates continue to support Sandbox.create after cache eviction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100