agent-substrate / agent-substrate/substrate

Locality-restricted resume failures report a misleading "no free workers available" error

Đang mở
#398 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/scheduling kind/bug
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

### Summary

When an actor's latest snapshot is a **local** checkpoint, `AssignWorkerStep.findFreeWorker` hard-restricts candidate workers to the node(s) listed in `LatestSnapshotInfo.Local.NodeVmsWithLocalSnapshots` (correctly — a local checkpoint only exists on the node that wrote it). But when that restriction is what eliminates all candidates, the returned error is the generic:

```
FailedPrecondition: no free workers available
```

This is misleading in both real failure modes:

- **The snapshot's node has no free workers** — free workers exist elsewhere in eligible pools, but they cannot restore the local checkpoint. The error reads as cluster-wide capacity exhaustion when it is actually a locality conflict.
- **The snapshot's node is gone** (scaled down, failed) — the actor's checkpointed state is unreachable, which is much more serious than "no free workers", and the operator gets no signal that this is the situation.

### Proposed fix

When no worker is picked **and** a node restriction was in effect, report a diagnostic error that distinguishes the cases, e.g.:

- `actor's local snapshot is on node(s) [gke-node-abc] but all 4 eligible workers on those nodes are busy; 12 free workers on other nodes cannot restore the local snapshot`
- `actor's local snapshot is on node(s) [gke-node-abc] but no eligible workers exist on those nodes (the node(s) may have been removed); 12 free workers on other nodes cannot restore the local snapshot`

The status code stays `FailedPrecondition`; only the message improves.

### Non-goals

This does not make the actor schedulable in either case. Real remediation (falling back to an external snapshot when one exists, snapshot placement tracking with liveness, load-shedding via snapshot replication) requires the snapshot data model to record external and local snapshot state simultaneously — today's `SnapshotInfo` oneof erases the external URI on pause. That redesign is a separate proposal; this issue only stops the error from pointing operators at the wrong problem.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.