atelet: a cold actor-image cache wedges request-driven resume the same way a cold sandbox-asset cache does, and per-node pre-warming does not fix it

### Expected Behavior

A request-driven resume onto a worker whose node has not yet unpacked the actor image should do one
of three things: pull the image off the request path so the resume succeeds; fail fast and
distinguishably so the caller can back off; or fail slowly but **converge**, so that retries
accumulate progress and a later attempt succeeds.

Relatedly: pre-warming a node should make it warm. If the documented way to prepare a pool reports
full coverage, a request-driven resume of a real actor onto that pool should work.

### Actual Behavior

None of the three. On a node whose **actor image** is not yet unpacked — sandbox assets already
cached, so #811 is not in play — the first request-driven `ResumeActor` fails, and **every retry
fails identically, with no cumulative progress**:

```
ResumeActor → workflow failed at step CallAteletRestore: while restoring durable snapshot:
while creating "agent" OCI bundle: in imageCache.EnsureImage:
while unpacking layer sha256:77286e45…:
while writing contents of "usr/libexec/gcc/x86_64-linux-gnu/14/lto1" from tar stream:
context deadline exceeded
```

Six attempts, each dying in the same large layer at **~5.5s elapsed** — the router's 5s
parked-request budget (`DefaultParkedRequestBudget`,
`cmd/atenet/internal/router/ingress/parking.go:29`). The gateway returned HTTP 500 after 23.8s and
the actor was left stranded in `ACTOR_STATE_RESUMING`.

The same image on the same nodes warms fine through the control plane, which carries no short
client deadline: ten concurrent `kubectl-ate resume actor` of the real template covered all ten
workers in **15.9s**, after which the identical request-driven turn succeeded in **12.2s** and a
follow-up in **3.2s**. So the unpack is not broken — its interaction with a short caller deadline
is. That is the same shape as #811, reached through a different cache.

And the pre-warm expectation fails silently: our coverage artefact reported **10/10 workers
warmed** immediately before the wedge above, because it was produced with a light stand-in
template. See "The pre-warm guidance trap" below.

### Steps to Reproduce the Problem

1. Bring up a pool of workers on nodes that have **never unpacked the actor image under test** — a
fresh cluster, or an existing one after a node-pool replacement. (Ours: 10 workers, 10 fresh
nodes.)
1. **Warm the sandbox-asset cache only**, so this reproduces #811's fix rather than #811 — boot any
light template through the ActorTemplate reconciler and let `ensureSandboxAssets` run to
completion on every node. Confirm `static-files` is populated. From here on the gVisor asset is
a cache hit and out of the picture.
1. Create an ActorTemplate whose image has **at least one layer larger than the router's 5s
parked-request budget can unpack**. Ours is a real agent image; the failing layer contains a
full gcc toolchain. Any image with a single fat layer will do.
1. Create an actor from that template and suspend it, so a durable snapshot exists and the next
turn takes the resume path.
1. Send a request that routes to the suspended actor, i.e. a **request-driven** resume through
`atenet-router`, not `kubectl-ate resume`. It fails at `CallAteletRestore` inside
`imageCache.EnsureImage` at ~5.5s.
1. **Retry, repeatedly.** Every attempt dies in the same layer at the same elapsed time. Check the
node: the layer's temp dir (`.tmp--*` alongside the layer pool) is gone after each
attempt, and the layer dir was never created. No progress accumulates, so the node never warms.
The actor sits in `ACTOR_STATE_RESUMING`.
1. Contrast, on the same node: `kubectl-ate resume actor` for the same template. It carries no
short deadline, completes, and leaves the unpacked layers behind. Every subsequent
request-driven resume on that node then succeeds.

Why CI does not catch it: e2e suites use small images whose every layer unpacks well inside the
budget, and single-node kind shares one image pool across all workers, so the second test in a
suite is always warm. The bug needs a fat layer **and** a node that has not seen it.

### Specifications

- **Version:** `4c1b37d` (the base of `release-0.1-rc`, and it contains #863 / `c9bccde5`).
Confirmed still unfixed on `main` at `23633f57` by reading the code — `internal/imagecache/` has
exactly one commit since `4c1b37d` (`85d404a8`) and it edits a README.
- **Platform:** GKE, gVisor sandbox class, 10 workers on 10 fresh nodes. Reproduced 27 Aug 2026.
- **Evidence:** pre-warm coverage artefacts for both templates (the light one and the real one),
the atelet log on `atelet-7gblv`, and the control-plane comparison above.

---

### Root cause

`internal/imagecache/imagecache.go`, on `main` at `23633f57`. Four mechanisms, and the first is the
one that makes it permanent.

**1. Progress is preserved per *completed* layer only.** `pull` writes the image record before any
unpack so every layer is referenced, and `cachedImage` re-pulls only what is missing — the code's
comment calls an interrupted pull's record "just resumable progress," and at layer granularity it
is. But `unpackLayerToPool` (`:577`) unpacks into a temp dir and renames at the end, cleaning the
temp dir on any error including cancellation, so that "a layer dir either exists complete or not at
all" (its own comment, `:575`). **A single layer that cannot finish inside the caller's budget
therefore restarts from zero on every attempt and never converges.** Six attempts on our cluster
made no cumulative progress whatsoever.

This is the same non-convergence @ygao-g identified for sandbox assets in
[#811 (comment)](https://github.com/agent-substrate/substrate/issues/811#issuecomment-5429270580)
after #863 — except here it was never self-healing to begin with, because the rename is at layer
granularity by design rather than by regression.

**2. One expiring layer cancels its siblings.** `pull` runs the layers under
`errgroup.WithContext` with `layerPullConcurrency = 4` (`:490`, `:96`), so the first layer to hit
the deadline cancels the other three in flight, discarding their partial temp dirs too. A pull that
was 3-of-4 layers along loses the in-flight work as well as the failing one.

**3. The singleflight propagates one caller's dying deadline to every waiter.** `EnsureImage`
collapses concurrent pulls of the same digest onto the winner (`:372`), and the comment is
explicit: *"The winning call's ctx governs the pull; if it is cancelled the waiters fail too."*
Under a burst onto a cold node — several actors of one template landing at once, which is exactly
what a fleet does — one caller's 5s budget expiring takes down every other caller waiting on the
same image, including any that had a longer budget.

**4. The caller's context reaches all of this unmodified.** `ensureAteletRestored`
(`cmd/ateapi/internal/controlapi/workflow_resume.go`) passes the inbound ctx straight into the
atelet `Restore` RPC, and `cmd/atelet/oci.go:114` hands it to `EnsureImage`. Nothing along that
path gives the unpack a budget of its own.

### Why the existing issues and the fix in flight do not cover this

- **#811** is the sandbox-asset extract. Its stack trace does contain `imageCache.EnsureImage`, but
it rules the image out and is right to: `after 0s, ctx err: context canceled`, zero bytes
fetched, the context already dead before the pull began. Ours is the inverse — assets warm,
deadline dies *inside* the unpack. A reader of #811 concludes the image cache is fine.
- **#1238** is the steady-state consequence of #811 (warm nodes track the number of goldens).
Warming a node with *a* golden does nothing for a different template's image.
- **#1358** pre-downloads assets named by `SandboxConfig`. The actor image is named by the
ActorTemplate, not the SandboxConfig, so #1358 structurally cannot reach it. **A node #1358 has
fully pre-warmed is still cold for every real agent image.**

The gVisor zstd work (google/gvisor#14528) makes this more visible rather than less: taking the
asset extract to ~2s removes the failure everyone is currently looking at and leaves this one, on a
slower cache, as the remaining way a fresh node wedges.

### The pre-warm guidance trap — worth fixing before any guidance ships

There are two cold caches, and warming with a light stand-in workload fills only the first. We
pre-warmed our own pool with a minimal actor on the reasoning that warming does not need an agent
and a lighter actor warms faster. The coverage artefact reported **10/10 workers warmed**. The very
next request-driven turn wedged, because the pool was warm for nothing anyone would actually run.

A coverage claim made with the wrong template is not incomplete, it is confidently wrong, and it
cost us a day of misdiagnosis. So:

- Any pre-warm guidance must say **per image**, and must not suggest a light stand-in workload.
- Any coverage artefact must record which template it was made with.
- As of `23633f57` there is **no pre-warm guidance anywhere in `docs/`** — a grep across `docs/`
and every `*.md` in the tree returns only vendored files. So there is currently nothing for
#1358 to amend, and a fresh install meets both wedges with nothing to find.

### Recovery, for anyone hitting this today

Resume the real template through the control plane once per node — `kubectl-ate resume actor`
carries no short deadline, completes, and leaves the unpacked layers behind for every later resume
on that node. Ten concurrent calls covered ten workers in 15.9s for us. Per **template**, not per
node. (For an actor already stranded in `ACTOR_STATE_RESUMING`, `DeleteActor` with `any_state` set
now works — per @zoez7 on #1238.)

### Candidate fixes

1. **Pre-warm by image, off the request path** — the equivalent of #1358 for the image cache, keyed
on the ActorTemplates a node's WorkerPools can schedule. Ideally the same mechanism, so there is
one answer to "is this node warm" rather than two.
1. **Make an interrupted unpack resumable within a layer**, so retries accumulate progress instead
of resetting. Weakest of the three — it fights the "complete or not at all" invariant, which
exists for good reasons.
1. **Fail fast and distinguishably when the image is not yet local** — an "image not yet local,
pull in progress" error the router can recognise and back off on, instead of burning the whole
budget on an unpack that will be cancelled. Pairs well with 1: the fast failure covers the
window before pre-warm completes.
1. **Do not let the singleflight propagate the winner's deadline to the waiters** — decouple the
pull's context from any single caller's, so a fleet burst does not have one short-budget request
killing the pull for everyone. Small and independently useful.

Related: #811, #1238, #1358, #863. None of them fix this.