Resume path: worker affinity has no failover, ~6s deadline cannot complete cold-cache restores, and ateom's image cache is unbounded/invisible to kubelet GC

The issue names ResumeActor, CallAteletRestore, RunWorkload, and imageCache.EnsureImage; begin by tracing those resume and bake entry points. Reproduce the cold-cache restore and worker-loss cases described, then inspect how deadlines, worker selection, and cache cleanup are handled. Done means the three reported failure modes have agreed behavior and can be verified against the scenarios in the report.

Observed on a live 3-node k3s cluster running upstream 4b12ce67 + the #818 fixes (fork sha 66310bb2), driving a real workload (a 690MB agent image) through the golden bake → suspend → ingress-resume lifecycle. Three related defects on the resume path; reporting together because they compound.

1. Resume is pinned to the checkpoint-holding worker with no failover

After a golden checkpointed on worker pod mfw74, the node's atelet was evicted (disk pressure). Every subsequent ResumeActor failed with:

workflow failed at step CallAteletRestore: for worker pod "agents/agent-smith-fleet-pool-...-mfw74":
selected atelet "ate-system/atelet-vgh9k" has no assigned IPs

— while a second, fully healthy idle worker sat in the same pool. The resume workflow retried the same dead placement indefinitely; it never considered the healthy worker. Snapshot storage is S3 (not worker-local), so a cross-worker restore appears mechanically possible.

2. The ResumeActor deadline (~6s observed) cannot complete a cold-cache restore, and retries cannot converge

With a fresh atelet (empty image cache), the restore required a full OCI unpack of the 690MB image. Each attempt died at ~5.5–6.4s:

while creating "agent" OCI bundle: in imageCache.EnsureImage:
while unpacking layer sha256:47eadb…: while writing contents of "usr/local/bin/…" from tar stream: context canceled
elapsed: 6.368s

Completed layers persist across attempts, so retries ratchet across layers — but any single layer needing >6s stalls the ratchet permanently (observed: three consecutive attempts dying at the same file in the same layer at the same ~5.5s mark). Each canceled attempt also leaves partial unpack data on disk (see 3).

By contrast RunWorkload (the bake path) completed the same unpack on the same pod with the same empty cache without issue — the deadline lives only on the resume path. Suggested: budget the resume deadline for image-ensure separately (or pre-warm via the bake path's cache), and/or make the deadline configurable.

3. ateom's image cache is not kubelet's image store

The cache is invisible to kubelet ImageGC (node images totaled 1.9GB while the node was 29GB over), unbounded, and lost whenever the atelet/worker pod is replaced. On a storage-tight node this is a failure amplifier: disk pressure evicts the atelet → cache resets to zero → every actor's next resume is a worst-case cold start whose canceled attempts write more debris to the same disk.

Environment

• k3s v1.36.1, 3 nodes, gVisor sandboxClass
• snapshot store: S3-compatible (SeaweedFS), path-style
• workload: real agent image (690MB, ~15 layers), golden bake ~75s, checkpoint 2.02s, snapshot 65.6MB

Happy to provide fuller logs or test patches.