ResumeActor returns Aborted "concurrent update conflict" when concurrent resumes pick the same worker

## Expected Behavior

`ResumeActor` should absorb transient optimistic-concurrency conflicts on the worker-assignment step and return either a resumed actor or a terminal error. A caller resuming an actor that nothing else is touching should not have to implement its own retry loop for a conflict ateapi created internally.

## Actual Behavior

Under sustained suspend/resume load on a 1,000-actor fleet, `ResumeActor` returns `codes.Aborted` with the message `"concurrent update conflict, please retry"` throughout the run. The bounded backoff in [`ensureWorkerAssigned`](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/ateapi/internal/controlapi/workflow_resume.go#L302-L351) [exhausts its five attempts](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/ateapi/internal/controlapi/workflow_resume.go#L319-L349) and returns a bare `store.ErrVersionConflict`, which `RPCService.ResumeActor` [maps to `codes.Aborted`](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/ateapi/internal/controlapi/actor.go#L424-L427).

The conflict is between concurrent resumes competing for the same worker. There is no reservation between the moment the scheduler picks a worker out of the cache and the moment the claim is written, so two calls can select the same worker at the same cached version and both write a conditional update predicated on it. One wins and the other is rejected.

This happens with the worker pool fully provisioned, so it is distinct from [`codes.ResourceExhausted`, `"no free workers available"`](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/ateapi/internal/controlapi/workflow_resume.go#L498-L506), which did not occur at all in the runs below. The scheduler always found a worker to pick; the failure is in claiming it.

## Workload

* 1,000 concurrent actors on a GKE cluster, worker pool sized 1:1, two `ate-api-server` replicas, PostgreSQL-backed store.
* Each actor loops for 10 minutes: `SuspendActor` over gRPC, then `ResumeActor` over gRPC **called directly against the control plane**, then three HTTP requests through the router (read 64 MiB, read it again warm, overwrite 64 MiB), each read verified with a SHA-256 digest.
* The `ActorTemplate` declares a `durableDir` volume with no external volume backing it, and `snapshotsConfig` `onPause: Full`, `onCommit: Full`.
* gVisor sandbox class.

The direct `ResumeActor` call is what makes this visible. [`ActorResumer` always treats `codes.Aborted` as retryable](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/atenet/internal/router/ingress/resumer.go#L142-L161) regardless of parking mode, and rides it out for [a 15 s budget when request parking is disabled](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/atenet/internal/router/ingress/resumer.go#L35-L38), or for the parking budget when it is enabled. Either way a resume driven through the router absorbs the conflict. Clients that call the control-plane API directly, including `kubectl-ate`, get the raw error.

## Evidence

Two consecutive 10-minute runs on one cluster, same commit and same image digest, with the two `ate-api-server` pods never restarted between them. The worker pool started the first run still filling and the second run already warm. Both runs produced aborts continuously once load was established, and the fleet did not converge out of the condition while load ran. Counts come from `ate-api-server` structured request logs over each run's 10-minute load window.

| | Run 1 | Run 2 |
| --- | ---: | ---: |
| Worker pool when load started | filling, 1 → 1,000 | warm, 1,000 ready |
| `ResumeActor` calls, all | 75,963 | 77,838 |
| of which are no-ops on already-running actors | 56,684 | 58,106 |
| **resumes that reached worker assignment** | **19,279** | **19,732** |
| `Aborted`, "concurrent update conflict" | 52 | 59 |
| Distinct actors affected | 50 | 59 |
| `ResourceExhausted`, "no free workers available" | 0 | 0 |

The no-op row is the router resuming an actor that is already running. Those return from the [pre-lease fast path](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/ateapi/internal/controlapi/workflow_resume.go#L93-L95) in a median of 0.3 ms, never take the actor lease, and cannot reach worker assignment, so they are separated out rather than mixed into the totals.

The remaining observations come from `"Picked worker"` records over the same windows.

**Two calls pick the same worker at the same cached version.** Grouping `"Picked worker"` records by the worker name and cache version they carry, a two-minute sample of one run's steady state contains several hundred clusters in which two or three *different* `ResumeActor` calls picked the same worker at the identical version. The picks within a cluster land within tens of milliseconds of each other. Each of those calls then issues an `UpdateWorker` carrying the same precondition, and only one can succeed.

**Calls that abort are contested far more often than calls in general.** Of the 4,448 picks in that sample, 848 were contested in this sense: another call picked the same worker at the same version within 40 ms. Among the picks made by calls that ended in `Aborted`, 37 of 37 were contested, and in every one of those the rival picked first. Other two-minute windows over the same run give the same picture, with the occasional uncontested pick. Contention is common across the board and close to universal among the failures, which is what distinguishes it from an incidental correlation.

**It is not only cross-replica cache divergence.** Roughly half of those rival picks came from the other `ate-api-server` replica and roughly half from the same one. A same-replica collision means one process handed the same cached worker to two concurrent RPCs, so the two caches disagreeing with each other is not a sufficient explanation on its own.

**Retries pick a different worker and collide again.** Retries almost always selected a worker the call had not already tried. The failure is therefore not a single stale cache entry being re-read, but repeated contention across the free set.

**A ready pool is not an idle pool.** Polling `kubectl get workerpool` every 10 seconds through both load windows returned `ready=1000` of `spec=1000` on every sample, and neither run logged `"no free workers available"`, the only route out of `scheduling.ErrNoCapacity`. That establishes a thousand healthy workers rather than a thousand available ones. With the pool sized 1:1, the set a resume can actually pick from holds only the workers whose actors are suspended at that instant, which is a small fraction of the fleet. Concurrent resumes are therefore drawing from a much smaller set than the pool size suggests.

**Every abort terminates on the retry budget.** All 111 aborts across the two runs completed in 168–283 ms server-side, with a p50 of 231 ms. That matches the `ensureWorkerAssigned` budget: with `wait.Backoff{Steps: 5, Duration: 10ms, Factor: 2.0, Jitter: 1.0}` the four sleeps are uniform over `[10,20)`, `[20,40)`, `[40,80)`, and `[80,160)` ms, totalling 150–300 ms. The band was computed from the code before it was looked for in the data. Every aborted call also shows more than one `"Picked worker"` record, never exactly one, which is consistent with the retry loop being what produced them.

The `Aborted` string is returned from several places in this workflow and the error itself does not identify which one fired, so the above does not pin the failure to a single line. What it does establish is that the calls which fail are the ones whose worker picks were contested, and that they fail after exhausting a retry loop rather than on a first attempt.

Two notes on scope. An attempt that [adopts a worker already carrying this actor's assignment](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/ateapi/internal/controlapi/workflow_resume.go#L466-L497) skips the pick and logs nothing, so the pick records above undercount attempts. And a terminal `Aborted` means all five attempts lost; conflicts resolved on attempts two through five surface nowhere, so contention is more common than the failure count suggests. Separately, run 1 lost a single actor to a crash, which produced 15 `FailedPrecondition` responses on later resumes of that actor. That is a different failure mode and is not included above.

## Observability Gap

Nothing on the server side counts this failure. [`schedulerRecordable`](https://github.com/agent-substrate/substrate/blob/39df566dfad8fe0226d1be0099be8666267b89b9/cmd/ateapi/internal/controlapi/workflow_resume.go#L423-L429) excludes version conflicts from the scheduler metric by design, and the worker-claim branch logs nothing on conflict, so retried attempts leave no trace in `ate.scheduler.assignment.duration`. The `Aborted` string is shared by 19 sites in `controlapi`. Three of them can return it during a single `ResumeActor`, and one of those three covers two distinct conflicts, since both the worker claim and the actor claim escape `ensureWorkerAssigned` as a bare `store.ErrVersionConflict` and are mapped at the same place. The error does not identify which step failed. The worker cache is already listed under `blind_spots` in `docs/metrics/substrate.yaml` for this shape of failure. Reconstructing the race meant correlating `"Picked worker"` records across traces, because no counter or log line records a lost claim.

## Steps to Reproduce the Problem

1. Bring up a cluster with a worker pool sized 1:1 with the actor count and more than one `ate-api-server` replica. Wait for the pool to report all workers ready before starting load.
2. Drive ~1,000 actors through a suspend/resume loop for 10 minutes, calling `ResumeActor` directly over gRPC rather than triggering resume through the router.
3. Collect `ResumeActor` failures carrying `codes.Aborted` with `"concurrent update conflict, please retry"`.
4. Check the server-side elapsed times of the aborted calls: they cluster in the 150–300 ms band the retry budget predicts.
5. Collect `"Picked worker"` records over the same window and group them by worker name and the version in the record. Clusters containing more than one trace ID are concurrent calls competing for one worker. Compare how often picks are contested across all calls against how often they are contested among the calls that aborted.

## Specifications

- Platform: GKE, gVisor (`runsc`) sandbox class, PostgreSQL-backed `ateapi` store
- Both runs were executed against `39df566d`, and all code references are permalinked to that commit