[Bug]: An actor whose worker pod goes away is permanently unrecoverable, and its clean snapshot cannot be used to bring it back

### What happened?

Any actor that is not already `SUSPENDED` when its worker pod disappears is moved to
`ACTOR_STATE_CRASHED` with its worker assignment cleared
(`cmd/ateapi/internal/controlapi/workflow_worker_delete.go`, `releaseBoundActor`).

**`CRASHED` is terminal.** Every verb that could bring the actor back is refused:

- `resume actor` → `FailedPrecondition: AssignWorker prerequisite not met (got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED or ACTOR_STATE_PAUSED)`
- `suspend actor` → `FailedPrecondition: MarkSuspending prerequisite not met (want ACTOR_STATE_RUNNING or ACTOR_STATE_PAUSED)`
- there is no `recover`, `restart` or `reset` verb

**And the actor still holds a clean snapshot that no API can start it from.** `status.latestSnapshot`
points at a complete snapshot in GCS, taken before the crash. The state is intact, addressable and
unusable. The only way out is to delete the actor and create a new one, losing everything it held.
Every subsequent request to it returns a 503 the caller cannot recover from.

We hit this by scaling a `WorkerPool` from 12 replicas back to 9. Two of ten actors happened to be
awake on removed pods. Both died in the same instant, 50 minutes before anyone looked at the
cluster.

**The documented Substrate operations are safe, and this report is scoped accordingly.**
`docs/upgrade.md` ground rule 2 says not to edit a serving worker pool, and runbook step c requires
every actor on a node to be suspended by hand before the node moves. A version roll done by the book
loses nothing, and our scale-down was the thing the ground rules tell you not to do. This is not a
report about the upgrade path.

**What is uncovered is infrastructure-level.** GKE node auto-upgrade, spot and preemptible nodes,
OOM kills and node auto-repair all reach `releaseBoundActor`, none is a Substrate verb, and none is
mentioned in any install prerequisite. Auto-upgrade is the one that matters: it is **on by default**
and it fires on the cloud provider's schedule rather than the operator's.

**`DrainWorker` is not a mitigation, which is worth stating explicitly.** It deliberately leaves the
resident actor in place; `TestDrainWorker_KeepsAssignment` asserts "the 1 left in place". Drain gates
new placement and does nothing for the actor already there, so an operator who reads drain as
"evacuate" is wrong.

**The pod shutdown path never checkpoints the actor.** Worker pods carry
`terminationGracePeriodSeconds: 3600`, so the intent to drain gracefully is clearly there. But
`AteomService.gracefulShutdown` cancels the in-flight RPC, waits for the lock, then SIGTERMs and
SIGKILLs the sandbox containers. It never asks the control plane to suspend the resident actor
first. The hour of grace is spent letting an *already running* snapshot finish rather than taking a
new one, which is why an actor that is merely awake, with no operation in flight, is guaranteed to
die rather than merely at risk.

### Expected Behavior

An actor should survive its worker pod going away, because worker pods going away is a routine
Kubernetes event rather than an exceptional one.

Ranked by coverage against cost, and the cheap one wins:

1. **Let `resume` accept a `CRASHED` actor when `status.latestSnapshot` is present.** This is the ask.
It covers every trigger, including preemption and OOM where no shutdown hook can help. It is a
precondition relaxation plus the restore path that already exists: that path restored these exact
snapshot objects 899 times without a failure in our acceptance soak, and the only thing refusing
it here is a state comparison. It converts permanent loss of the actor into loss of the turn that
was in flight, which the caller already knows it lost, and there is no double-execution risk
because the snapshot predates the turn. An explicit recover verb would do as well if you prefer
not to widen `resume`'s preconditions; that is your API call, not ours.
2. **Checkpoint the resident actor on drain**, which is what the 3600s grace period is already sized
for. Cleaner, because no turn is lost, but a bigger change and it still cannot cover preemption or
OOM. Worth doing after (1), not instead of it.
3. **Not worth asking for:** refusing to remove a worker that holds an awake actor. Cheap, but a
kubelet eviction ignores it, so it only catches the trigger you already document against.

**Separately from the code, the install prerequisites should say that worker node pools need
auto-upgrade off and must not be spot**, alongside the Workload Identity and beta-API prerequisites
already documented. That closes the only trigger that fires without an operator deciding anything,
and it is the part that matters for the current release. It is not a substitute for (1): telling
people to disable a standard Kubernetes safety feature is a reasonable first-release answer and not
a reasonable 1.0 one.

### Steps to Reproduce

```
1. Create a WorkerPool with 2 replicas and an ActorTemplate for any workload.

2. Create an actor and drive one request into it so that it is RUNNING and
resident on a worker:

kubectl-ate create actor -a agents actor-1 --template-ref agent
curl -sS "$INGRESS" -d '{"prompt":"hello"}'
kubectl-ate get actor -a agents actor-1 -o json | jq .status.state
=> ACTOR_STATE_RUNNING

3. Note which worker it is on, and that it has a good snapshot:

kubectl-ate get actor -a agents actor-1 -o json \
| jq '{worker: .status.workerAssignment, snap: .status.latestSnapshot}'

4. Remove that worker pod. Scaling the pool down is the gentlest way:

kubectl -n ate-system scale deploy/ --replicas=1

(a node upgrade, a pod delete, or an OOM kill all reach the same path)

5. Observe the actor:

kubectl-ate get actor -a agents actor-1 -o json | jq .status
=> state: ACTOR_STATE_CRASHED
=> workerAssignment: absent, cleared by the same update
=> latestSnapshot: still present, still complete

6. Try every recovery verb:

kubectl-ate resume actor -a agents actor-1 # FailedPrecondition
kubectl-ate suspend actor -a agents actor-1 # FailedPrecondition
=> no verb exists that starts the actor from the snapshot it still holds.
```

### Sandbox Runtime

gVisor (runsc)

### Agent Substrate Version / Commit SHA

`c48b3a3c` (`release-0.1`)

### Kubernetes Version & Environment

GKE 1.35 (`v1.35.7-gke.1150000`), 5 nodes, us-central1-c

### Host OS & Architecture

Linux 6.6 (x86_64)

### Relevant Logs and Diagnostic Output

```shell
# Both actors, captured after a WorkerPool 12 -> 9 scale-down.
# Sanitized: atespace, actor names, bucket.
{
"actors": [
{
"actorTemplate": {"atespace": "agents", "name": "agent"},
"metadata": {
"atespace": "agents", "name": "actor-1",
"uid": "55f472ad-34fd-4f5e-ab52-317d74d8e664",
"createTime": "2026-09-05T18:21:56.983654827Z",
"updateTime": "2026-09-06T17:30:21.414820654Z",
"version": "20"
},
"status": {
"state": "ACTOR_STATE_CRASHED",
"latestSnapshot": {
"atespace": "gs://REDACTED-snapshots/.../actors/55f472ad-.../snapshots/7c626280-..."
}
}
},
{
"actorTemplate": {"atespace": "agents", "name": "agent"},
"metadata": {
"atespace": "agents", "name": "actor-2",
"uid": "24f7280d-44c8-4dbb-a7f3-4c5e2c6074ce",
"createTime": "2026-09-06T16:33:06.474349398Z",
"updateTime": "2026-09-06T17:30:21.317997293Z",
"version": "12"
},
"status": {
"state": "ACTOR_STATE_CRASHED",
"latestSnapshot": {
"atespace": "gs://REDACTED-snapshots/.../actors/24f7280d-.../snapshots/13fb5073-..."
}
}
}
]
}

# Both updateTime values are within 100ms of each other and of the scale-down.
# Note what is NOT in either document: no workerAssignment (cleared by the same
# update that set CRASHED, so nothing downstream can say where the actor was or
# fetch its logs), and no reason field, so CRASHED never says why.

# Recovery attempts
$ kubectl-ate resume actor -a agents actor-1
Error: rpc error: code = FailedPrecondition desc = AssignWorker prerequisite not met
(got: ACTOR_STATE_CRASHED, want ACTOR_STATE_SUSPENDED or ACTOR_STATE_PAUSED)

$ kubectl-ate suspend actor -a agents actor-1
Error: rpc error: code = FailedPrecondition desc = MarkSuspending prerequisite not met
(want ACTOR_STATE_RUNNING or ACTOR_STATE_PAUSED)
```

### Additional Context

**Two smaller fixes belong with this one**, both visible in the evidence above:

- The crash clears `status.workerAssignment` in the same update that sets `CRASHED`, so after the
fact nothing can tell which worker the actor was on and no log can be retrieved. Keeping the last
assignment on a crashed actor costs nothing and makes the failure investigable.
- The actor document carries no reason field, so `CRASHED` never says why. An operator seeing a
crashed actor cannot distinguish this from any other crash cause.

**This issue also closes a second one**: an untolerated snapshot-GC delete in the suspend workflow
can wedge an actor in `SUSPENDING`, which then drains into `CRASHED`. That wedge is only data loss
because `CRASHED` is terminal, so fix (1) above and it becomes a retryable stuck suspend with
nothing further owed. Cross-link the two rather than working them separately.

**Diagnosing this after the fact is currently very hard**, worth noting for anyone reproducing: the
api-server logs full request and response bodies at INFO, including an entire `ListActors` dump per
poll. A 3-hour log query reaches back roughly 16 minutes of wall time, so our crash window had
already aged out of a query we ran within the hour.

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [ ] I have verified that this issue occurs on the latest commit on `main`.