agent-substrate / agent-substrate/substrate
Locality-restricted resume failures report a misleading "no free workers available" error
- Linguagem predominante
- Go
- Estrelas
- 1.8k
- Forks
- 316
- Merge médio
- 2d 43min
- PRs com merge (30d)
- 287
Descrição
### Summary
When an actor's latest snapshot is a **local** checkpoint, `AssignWorkerStep.findFreeWorker` hard-restricts candidate workers to the node(s) listed in `LatestSnapshotInfo.Local.NodeVmsWithLocalSnapshots` (correctly — a local checkpoint only exists on the node that wrote it). But when that restriction is what eliminates all candidates, the returned error is the generic:
```
FailedPrecondition: no free workers available
```
This is misleading in both real failure modes:
- **The snapshot's node has no free workers** — free workers exist elsewhere in eligible pools, but they cannot restore the local checkpoint. The error reads as cluster-wide capacity exhaustion when it is actually a locality conflict.
- **The snapshot's node is gone** (scaled down, failed) — the actor's checkpointed state is unreachable, which is much more serious than "no free workers", and the operator gets no signal that this is the situation.
### Proposed fix
When no worker is picked **and** a node restriction was in effect, report a diagnostic error that distinguishes the cases, e.g.:
- `actor's local snapshot is on node(s) [gke-node-abc] but all 4 eligible workers on those nodes are busy; 12 free workers on other nodes cannot restore the local snapshot`
- `actor's local snapshot is on node(s) [gke-node-abc] but no eligible workers exist on those nodes (the node(s) may have been removed); 12 free workers on other nodes cannot restore the local snapshot`
The status code stays `FailedPrecondition`; only the message improves.
### Non-goals
This does not make the actor schedulable in either case. Real remediation (falling back to an external snapshot when one exists, snapshot placement tracking with liveness, load-shedding via snapshot replication) requires the snapshot data model to record external and local snapshot state simultaneously — today's `SnapshotInfo` oneof erases the external URI on pause. That redesign is a separate proposal; this issue only stops the error from pointing operators at the wrong problem.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.