[Bug]: An actor whose resume, suspend or delete fails gets stuck in RESUMING / SUSPENDING / DELETING; it cannot be deleted, its worker stays allocated, and no API call recovers it

Start with hack/repro/stuck-actor/README and run repro_wedge.py followed by recover_probe.py to reproduce the stuck lifecycle. Trace the ResumeActor, SuspendActor, and DeleteActor paths, especially their runsc teardown handling. Done means a node-side failure leaves the actor recoverable through DeleteActor(any_state=true), releases the worker, and handles a missing runsc container as already torn down.

What happened?

When a lifecycle operation fails on the node, the actor is left in the transitional state it was already committed to and never leaves it:

• a failed resume leaves it ACTOR_STATE_RESUMING
• a failed suspend leaves it ACTOR_STATE_SUSPENDING
• a failed delete leaves it ACTOR_STATE_DELETING

In all three the worker keeps allocated.actors = 1. Nothing in the API gets it out. Re-issuing the same operation returns the same error every time, and DeleteActor(any_state=true) fails on the node teardown and leaves the actor DELETING, still allocated. The only recovery today is deleting the worker pod, which moves the actor to CRASHED and frees the worker.

How we saw it happen

A few ways we saw to reproduce it (and happened in benchmarks):

1. The request is cancelled or times out mid-teardown. ateom runs runsc kill/wait/state/delete under the request context; a cancellation landing mid-runsc delete SIGKILLs runsc partway through destroying the container.
2. runsc delete fails. runsc delete -force can remove the container's state record and still exit 128 (e.g. 5s timeout). Every later runsc state/runsc delete on that container then fatal-exits 128 (loading container: file does not exist), so the teardown can never complete on retry.
3. The sandbox is lost. Observed as the ateom container being OOM-killed, for example. runsc checkpoint then fails (cannot checkpoint container "_pause" in state stopped) and the delete hits (2).

Expected Behavior

An actor stuck in a transitional state after a node-side failure must be recoverable through the API:

• DeleteActor(any_state=true) should succeed and free the worker. A container runsc no longer has a record of is already gone and teardown should treat it as done.
• Better: the actor should be moved to CRASHED and its worker released, the way a missing worker pod already is.

Steps to Reproduce

Scripts (vibe coded): hack/repro/stuck-actor (talk to ateapi and the router directly; setup in the README there).

python3 repro_wedge.py oomhog --template <template>
# -> VERDICT[oomhog]: PINNED (actor=SUSPENDING, workers_allocated=[...])
python3 recover_probe.py <actor>
# -> DeleteActor(any_state=true) -> INTERNAL ... runsc delete: exit status 128
#    +20s state: DELETING | workers allocated: 1

By hand:

1. Create an actor from any template and resume it.
2. Kill the sandbox (e.g. allocate past the worker's memory limit; the ateom container is OOM-killed and restarted).
3. SuspendActor returns INTERNAL (runsc checkpoint: exit status 128); the actor is SUSPENDING, worker allocated.
4. DeleteActor(any_state=true) returns INTERNAL (runsc delete: exit status 128); the actor flips to DELETING and stays; worker still allocated. Repeating any call gives the same result.

Sandbox Runtime

gVisor (runsc)