agent-substrate / agent-substrate/substrate
[P2] Worker slot leaked when ateapi crashes between UpdateWorker and UpdateActor
- 主要語言
- Go
- 星號
- 1.8k
- 分支
- 316
- 平均合併
- 2 天 43 分鐘
- 30 天內合併 PR
- 287
描述
**Severity:** P2 (self-heals on user retry or pod death; reduces pool capacity in interim)
**Component:** Control Plane — `cmd/ateapi/internal/controlapi/workflow_resume.go`
**Audit ID:** CP-1
**Maps to suspect:** S1
---
## Summary
`AssignWorkerStep.Execute` claims a worker by calling `store.UpdateWorker` (sets
`Assignment`) before calling `store.UpdateActor` (sets `STATUS_RESUMING`). If ateapi
crashes between these two writes, the worker record has an `Assignment` pointing at the
actor but the actor is still `STATUS_SUSPENDED`. No background reclaimer detects this
stale claim. Pool capacity is reduced by one until the user retries `ResumeActor` (which
triggers the recovery loop) or the worker pod is deleted.
---
## Impact
- Worker pool capacity silently reduced by 1 per occurrence.
- In a busy cluster with frequent user abandonment of actors, this drains the pool
over time.
- No monitoring metric or alert for "worker claimed but actor not resuming."
---
## Root Cause
**File:** `cmd/ateapi/internal/controlapi/workflow_resume.go` lines 218–250
```go
// Line 218: claims the worker
if err := store.UpdateWorker(..., Assignment: actorUID); err != nil { ... }
// --- CRASH WINDOW ---
// Line 229: marks actor RESUMING
if err := store.UpdateActor(..., Status: STATUS_RESUMING); err != nil { ... }
```
The recovery loop at lines 165–193 fires only at the start of a new `ResumeActor` RPC,
scanning for a worker whose `Assignment == actorUID` when the actor is still SUSPENDED.
If the user never retries, the claim persists until pod death.
---
## Steps to Reproduce
1. Have a `STATUS_SUSPENDED` actor and a free worker pool.
2. Call `ResumeActor`, kill ateapi after `UpdateWorker` returns but before `UpdateActor`:
```bash
kubectl ate resume actor my-actor -a demo &
sleep 0.3 # enough for UpdateWorker to complete
kubectl delete pod -n ate-system -l app=ate-api-server
```
3. After ateapi restarts, list workers:
```bash
kubectl ate list workers # or inspect Valkey directly
# Worker has Assignment=my-actor but actor is STATUS_SUSPENDED
```
4. Do NOT retry ResumeActor. Observe the worker is never returned to the free pool.
---
## Expected Behavior
A background goroutine (or a startup scan) should detect workers with `Assignment` set
pointing at an actor that is still `STATUS_SUSPENDED`/`STATUS_PAUSED`, and clear the
assignment automatically.
---
## Suggested Fix
In `reconcileOrphanedWorkers` (already runs at startup), also scan for workers where
`Assignment != nil` but the referenced actor has status `SUSPENDED` or `PAUSED`. Clear
the assignment. This closes the window without needing a two-phase commit.
貢獻指南
評估
這個 Issue 還沒有評估資料。