agent-substrate / agent-substrate/substrate

Cold-node gVisor release extract (~20s, silent, uncancellable) exceeds the router resume budget — first resume after a release bump always fails

Abierto
#811 13 comentarios 0 reacciones 1 asignado Reclamado por @nybidari Ver en GitHub
area/gvisor kind/bug
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

## Symptom

On a node whose gVisor asset cache is cold, the first `ResumeActor` reliably fails with:

```
workflow failed at step CallAteletRestore: while restoring durable snapshot:
rpc error: code = Canceled desc = context canceled
```

and atelet's `Restore` reports (with newly added pull logging):

```
while creating "counter" OCI bundle: in imageCache.EnsureImage:
in remote.Image (registry "gcr.io", gcp_auth=true, after 0s, ctx err: context canceled):
Get "https://gcr.io/v2/": context canceled elapsed-time: 19.9s
```

The actor image is not the problem — zero bytes of it were fetched (`after 0s`); the context was already dead when the pull began.

## Timeline (GKE, `demo/my-counter-4`, trace `96d4b7e35eee327b1ec90a0dc3950df6`)

| t | event |
|---|---|
| 15:51:01.42–01.44 | snapshot files downloaded + zstd-decompressed (5.2MB total, <60ms) |
| ~15:51:06 | caller (atenet-router, ~5s resume deadline) cancels the context |
| 15:51:01.44–15:51:21.17 | **19.7s silent gap**: `ensureSandboxAssets` downloads `gs://gvisor/releases/release/20260803/x86_64/gvisor.tar.bz2` and extracts it via `extractTarBz2` |
| 15:51:21.17 | prep leg reaches the OCI image pull; first HTTP request dies instantly on the canceled context |

## Root cause

`cmd/atelet/sandbox_assets.go` `fetchGVisorRelease` → `extractTarBz2` (introduced in #684, when the gVisor asset became a release tarball instead of a bare `runsc` binary):

- **Silent**: no log lines for the download or the extraction — a 20s hole in the trace.
- **Uncancellable**: the bzip2+tar extraction loop has no `ctx` checks (Go's stdlib bzip2 is single-threaded and slow), so it runs ~15s past the caller's cancellation.
- **Fleet-wide cold**: #787 bumped the pinned release, invalidating the content-addressed cache (`StaticFilesDir`, hostPath) on every node at once — so every node's *first* resume after the rollout hits this.

#684's benchmarks measure warm-asset-cache steady state (which got faster); the per-node, per-release cold extraction cost was invisible to them.

Interacting existing issues: #606 (router retry budget), #646 (`Canceled` strands actors in RESUMING/SUSPENDING).

## Mitigating behavior (why it self-heals)

The extraction completes and renames into the content-addressed release dir even though the RPC's caller is gone, so the node pays the cost once; the next resume on that node finds the assets cached and succeeds. Each node fails exactly one restore per release bump (more if the actor isn't retried — the actor stays `RESUMING` until another request retries it).

## Candidate fixes (roughly independent)

1. **Observability**: log start/size/duration of asset downloads and the tarball extraction in `ensureSandboxAssets`/`fetchGVisorRelease`, so the phase is attributable from logs (the restore's phase metrics do record `SnapshotPhaseSandboxAssets`, but nothing lands in the trace).
2. **Pre-warm off the request path**: have atelet pre-fetch/extract pinned gVisor releases at startup (from the node's SandboxConfigs), so a release bump costs the node at rollout time, not on the first user resume.
3. **ctx checks in `extractTarBz2`/`downloadVerified`**: fail fast on cancellation. Trade-off: today the cancelled extract still populates the cache, which is what makes the retry succeed — aborting early would make retries pay the full cost again unless the extraction is detached from the request context instead (see 4).
4. **Detach node-local cache population from the caller's deadline**: run asset extraction (and image pulls, cf. the layer pool) under a server-side context so a hung-up caller doesn't abort work whose product is a shared cache. Related to the broader deadline discussion in #606/#646.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.