agent-substrate / agent-substrate/substrate

Locality-restricted resume failures report a misleading "no free workers available" error

Open
#398 0 comments 0 reactions 0 assignees View on GitHub
area/scheduling kind/bug
Dominant language
Go
Stars
1.8k
Forks
316
Avg merge
2d 43m
Merged PRs (30d)
287

Description

### Summary

When an actor's latest snapshot is a **local** checkpoint, `AssignWorkerStep.findFreeWorker` hard-restricts candidate workers to the node(s) listed in `LatestSnapshotInfo.Local.NodeVmsWithLocalSnapshots` (correctly — a local checkpoint only exists on the node that wrote it). But when that restriction is what eliminates all candidates, the returned error is the generic:

```
FailedPrecondition: no free workers available
```

This is misleading in both real failure modes:

- **The snapshot's node has no free workers** — free workers exist elsewhere in eligible pools, but they cannot restore the local checkpoint. The error reads as cluster-wide capacity exhaustion when it is actually a locality conflict.
- **The snapshot's node is gone** (scaled down, failed) — the actor's checkpointed state is unreachable, which is much more serious than "no free workers", and the operator gets no signal that this is the situation.

### Proposed fix

When no worker is picked **and** a node restriction was in effect, report a diagnostic error that distinguishes the cases, e.g.:

- `actor's local snapshot is on node(s) [gke-node-abc] but all 4 eligible workers on those nodes are busy; 12 free workers on other nodes cannot restore the local snapshot`
- `actor's local snapshot is on node(s) [gke-node-abc] but no eligible workers exist on those nodes (the node(s) may have been removed); 12 free workers on other nodes cannot restore the local snapshot`

The status code stays `FailedPrecondition`; only the message improves.

### Non-goals

This does not make the actor schedulable in either case. Real remediation (falling back to an external snapshot when one exists, snapshot placement tracking with liveness, load-shedding via snapshot replication) requires the snapshot data model to record external and local snapshot state simultaneously — today's `SnapshotInfo` oneof erases the external URI on pause. That redesign is a separate proposal; this issue only stops the error from pointing operators at the wrong problem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.