test(resilience): verify resume works — auto-restart on abort + alerting

Motivation

Long V1/V2 runs take 8–20 hours per (model × harness) in our recent runs (per wall_hours in leaderboard/results.csv). Today's failure modes that show up in the wild:

• Container OOM — Chromium balloons mid-run, kernel kills it, batch aborts at task N/130
• Network blip — Anthropic/OpenAI/OpenRouter 502 for 30s, agent times out, batch reports task as failed (but really it was an infra issue, not the agent's)
• Site captcha mid-run — agent gets through 60 tasks then hits a captcha on task 61, blocked for the rest
• Host VM evicted — Vultr / cloud reboot in the middle of a multi-hour batch
• Queue cap — claude-opus-4-7 hit our queue cap at task 75 (partial), gpt-5.5 at 81 (partial) — should those have auto-resumed when the cap reset?

Right now the script just stops, leaving a (model, harness, corpus) cell partial on the leaderboard ("partial" tag on 2/6 V2 rows already). Every partial is a small operator-pain point: someone has to notice, manually restart, and re-merge results.

This issue asks: prove resume works end-to-end, then add auto-restart + alerting where it doesn't.

What to test

Test 1 — Manual resume mid-batch

# Start a V2 hermes run on a cheap model
clawbench run --corpus v2 --model deepseek-v4-flash --harness hermes &
PID=$!
sleep 600          # 10 min in — should be ~5–10 tasks deep
kill $PID          # simulate abort
clawbench run --corpus v2 --model deepseek-v4-flash --harness hermes  # re-invoke

Expected: the second invocation picks up where the first left off (doesn't re-run tasks 1–10). Verify by:

• Counting trace dirs under claw-output/v2-results/deepseek-v4-flash_hermes_v2/ — should be ≥10 after the kill, then continue from there
• results.csv post-completion should reflect exactly 130 tasks, not 140

Test 2 — Mid-task abort

Kill while task #42 is mid-execution (agent has been clicking around for 3 minutes). Re-invoke. Does the resume:

• (a) Skip task #42 (assume done) — wrong, lost data
• (b) Re-run task #42 from scratch — expected
• (c) Try to resume within task #42 — only correct if we have action-level checkpointing (probably no, document the limitation)

Test 3 — Auto-restart on abort

Add a wrapper:

clawbench run --corpus v2 --model X --harness hermes --auto-restart 3

If the batch aborts (process exits non-zero), wait 30s, re-invoke up to 3 times. Each retry resumes from the prior checkpoint (per Test 1). Final exit code is success iff total attempted-task-count reaches the corpus size.

Test 4 — Alerting

Wire up clawbench-eval-watch (new subcommand) that polls the running batch dir every 60s and:

• Posts to a Slack/Discord webhook on abort (with last-task-id, error tail, runtime so far)
• Posts on completion (with final pass/total/wall_hours)
• Optionally posts a heartbeat every N tasks ("deepseek-v4-flash · 50/130 done · 1.8h elapsed · 9 passes so far")

Configurable via ~/.config/clawbench/notify.toml so each operator can route their own.

Implementation hints

• Checkpoint state already exists per-task as the run dir (<corpus>-<task_id>-<task_slug>-<model>-<timestamp>/). Resume = scan the batch dir, find the highest task_id with .sync_complete, start from the next.
• Time-out / abort detection — clawbench run already returns non-zero on abort; the --auto-restart wrapper is the new piece.
• Alerting webhook — small POST to a configurable URL; reuse the existing requests dep, no new packages.
• Idempotency — if a partial run dir exists for a task (e.g., .recording_in_progress but no .sync_complete), the resume should delete that dir and re-run that task, not skip.

Acceptance criteria

• Test 1 passes — manual kill mid-batch resumes correctly, no task duplicated or skipped
• Test 2 documented — clear behavior on mid-task abort (rerun the task, lose the partial in-progress data)
• --auto-restart N flag implemented + smoke-tested with synthetic abort
• clawbench-eval-watch skeleton subcommand that posts to a webhook on abort/completion (heartbeat optional)
• docs/operations.md updated with: "How to resume a run", "How to set up alerting", "What survives an abort and what doesn't"

Why this matters now

We have 2 partial V2 batches (claude-opus-4-7 75/130, gpt-5.5 81/130) on the live leaderboard right now, both labeled "partial" because they hit queue caps mid-run. Auto-restart would have re-queued them when the cap reset; we'd have full data instead of partial. Same will happen on the next round of new models. The earlier we cement resume + retry semantics, the fewer partial cells we ship.

Out of scope

• Distributed / multi-machine batch resumption — single-host only
• Resume within a task (action-level checkpointing) — too invasive for this issue
• Replacing the existing run script with a daemon — keep CLI-driven for now

Start at the clawbench run and clawbench-eval-watch entry points, then inspect the run directory layout described in the issue, including .sync_complete and .recording_in_progress. Reproduce the manual and mid-task abort scenarios and verify results.csv has exactly 130 tasks without duplicates. Done means the acceptance checklist passes, including --auto-restart, webhook notifications, and docs/operations.md updates.