agent-substrate / agent-substrate/substrate
Cold-node gVisor release extract (~20s, silent, uncancellable) exceeds the router resume budget — first resume after a release bump always fails
- Linguagem predominante
- Go
- Estrelas
- 1.8k
- Forks
- 316
- Merge médio
- 2d 43min
- PRs com merge (30d)
- 287
Descrição
## Symptom
On a node whose gVisor asset cache is cold, the first `ResumeActor` reliably fails with:
```
workflow failed at step CallAteletRestore: while restoring durable snapshot:
rpc error: code = Canceled desc = context canceled
```
and atelet's `Restore` reports (with newly added pull logging):
```
while creating "counter" OCI bundle: in imageCache.EnsureImage:
in remote.Image (registry "gcr.io", gcp_auth=true, after 0s, ctx err: context canceled):
Get "https://gcr.io/v2/": context canceled elapsed-time: 19.9s
```
The actor image is not the problem — zero bytes of it were fetched (`after 0s`); the context was already dead when the pull began.
## Timeline (GKE, `demo/my-counter-4`, trace `96d4b7e35eee327b1ec90a0dc3950df6`)
| t | event |
|---|---|
| 15:51:01.42–01.44 | snapshot files downloaded + zstd-decompressed (5.2MB total, <60ms) |
| ~15:51:06 | caller (atenet-router, ~5s resume deadline) cancels the context |
| 15:51:01.44–15:51:21.17 | **19.7s silent gap**: `ensureSandboxAssets` downloads `gs://gvisor/releases/release/20260803/x86_64/gvisor.tar.bz2` and extracts it via `extractTarBz2` |
| 15:51:21.17 | prep leg reaches the OCI image pull; first HTTP request dies instantly on the canceled context |
## Root cause
`cmd/atelet/sandbox_assets.go` `fetchGVisorRelease` → `extractTarBz2` (introduced in #684, when the gVisor asset became a release tarball instead of a bare `runsc` binary):
- **Silent**: no log lines for the download or the extraction — a 20s hole in the trace.
- **Uncancellable**: the bzip2+tar extraction loop has no `ctx` checks (Go's stdlib bzip2 is single-threaded and slow), so it runs ~15s past the caller's cancellation.
- **Fleet-wide cold**: #787 bumped the pinned release, invalidating the content-addressed cache (`StaticFilesDir`, hostPath) on every node at once — so every node's *first* resume after the rollout hits this.
#684's benchmarks measure warm-asset-cache steady state (which got faster); the per-node, per-release cold extraction cost was invisible to them.
Interacting existing issues: #606 (router retry budget), #646 (`Canceled` strands actors in RESUMING/SUSPENDING).
## Mitigating behavior (why it self-heals)
The extraction completes and renames into the content-addressed release dir even though the RPC's caller is gone, so the node pays the cost once; the next resume on that node finds the assets cached and succeeds. Each node fails exactly one restore per release bump (more if the actor isn't retried — the actor stays `RESUMING` until another request retries it).
## Candidate fixes (roughly independent)
1. **Observability**: log start/size/duration of asset downloads and the tarball extraction in `ensureSandboxAssets`/`fetchGVisorRelease`, so the phase is attributable from logs (the restore's phase metrics do record `SnapshotPhaseSandboxAssets`, but nothing lands in the trace).
2. **Pre-warm off the request path**: have atelet pre-fetch/extract pinned gVisor releases at startup (from the node's SandboxConfigs), so a release bump costs the node at rollout time, not on the first user resume.
3. **ctx checks in `extractTarBz2`/`downloadVerified`**: fail fast on cancellation. Trade-off: today the cancelled extract still populates the cache, which is what makes the retry succeed — aborting early would make retries pay the full cost again unless the extraction is detached from the request context instead (see 4).
4. **Detach node-local cache population from the caller's deadline**: run asset extraction (and image pulls, cf. the layer pool) under a server-side context so a hung-up caller doesn't abort work whose product is a shared cache. Related to the broader deadline discussion in #606/#646.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.