agent-substrate / agent-substrate/substrate

Locality-restricted resume failures report a misleading "no free workers available" error

未關閉
#398 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/scheduling kind/bug
主要語言
Go
星號
1.8k
分支
316
平均合併
2 天 43 分鐘
30 天內合併 PR
287

描述

### Summary

When an actor's latest snapshot is a **local** checkpoint, `AssignWorkerStep.findFreeWorker` hard-restricts candidate workers to the node(s) listed in `LatestSnapshotInfo.Local.NodeVmsWithLocalSnapshots` (correctly — a local checkpoint only exists on the node that wrote it). But when that restriction is what eliminates all candidates, the returned error is the generic:

```
FailedPrecondition: no free workers available
```

This is misleading in both real failure modes:

- **The snapshot's node has no free workers** — free workers exist elsewhere in eligible pools, but they cannot restore the local checkpoint. The error reads as cluster-wide capacity exhaustion when it is actually a locality conflict.
- **The snapshot's node is gone** (scaled down, failed) — the actor's checkpointed state is unreachable, which is much more serious than "no free workers", and the operator gets no signal that this is the situation.

### Proposed fix

When no worker is picked **and** a node restriction was in effect, report a diagnostic error that distinguishes the cases, e.g.:

- `actor's local snapshot is on node(s) [gke-node-abc] but all 4 eligible workers on those nodes are busy; 12 free workers on other nodes cannot restore the local snapshot`
- `actor's local snapshot is on node(s) [gke-node-abc] but no eligible workers exist on those nodes (the node(s) may have been removed); 12 free workers on other nodes cannot restore the local snapshot`

The status code stays `FailedPrecondition`; only the message improves.

### Non-goals

This does not make the actor schedulable in either case. Real remediation (falling back to an external snapshot when one exists, snapshot placement tracking with liveness, load-shedding via snapshot replication) requires the snapshot data model to record external and local snapshot state simultaneously — today's `SnapshotInfo` oneof erases the external URI on pause. That redesign is a separate proposal; this issue only stops the error from pointing operators at the wrong problem.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。