agent-substrate / agent-substrate/substrate

[P2] Worker slot leaked when ateapi crashes between UpdateWorker and UpdateActor

オープン
#607 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
area/api-machinery area/scheduling kind/bug prio/P2
主要言語
Go
スター
1.8k
フォーク
316
平均マージ
2日 43分
マージ済み PR(30日)
287

説明

**Severity:** P2 (self-heals on user retry or pod death; reduces pool capacity in interim)
**Component:** Control Plane — `cmd/ateapi/internal/controlapi/workflow_resume.go`
**Audit ID:** CP-1
**Maps to suspect:** S1

---

## Summary

`AssignWorkerStep.Execute` claims a worker by calling `store.UpdateWorker` (sets
`Assignment`) before calling `store.UpdateActor` (sets `STATUS_RESUMING`). If ateapi
crashes between these two writes, the worker record has an `Assignment` pointing at the
actor but the actor is still `STATUS_SUSPENDED`. No background reclaimer detects this
stale claim. Pool capacity is reduced by one until the user retries `ResumeActor` (which
triggers the recovery loop) or the worker pod is deleted.

---

## Impact

- Worker pool capacity silently reduced by 1 per occurrence.
- In a busy cluster with frequent user abandonment of actors, this drains the pool
over time.
- No monitoring metric or alert for "worker claimed but actor not resuming."

---

## Root Cause

**File:** `cmd/ateapi/internal/controlapi/workflow_resume.go` lines 218–250

```go
// Line 218: claims the worker
if err := store.UpdateWorker(..., Assignment: actorUID); err != nil { ... }
// --- CRASH WINDOW ---
// Line 229: marks actor RESUMING
if err := store.UpdateActor(..., Status: STATUS_RESUMING); err != nil { ... }
```

The recovery loop at lines 165–193 fires only at the start of a new `ResumeActor` RPC,
scanning for a worker whose `Assignment == actorUID` when the actor is still SUSPENDED.
If the user never retries, the claim persists until pod death.

---

## Steps to Reproduce

1. Have a `STATUS_SUSPENDED` actor and a free worker pool.
2. Call `ResumeActor`, kill ateapi after `UpdateWorker` returns but before `UpdateActor`:
```bash
kubectl ate resume actor my-actor -a demo &
sleep 0.3 # enough for UpdateWorker to complete
kubectl delete pod -n ate-system -l app=ate-api-server
```
3. After ateapi restarts, list workers:
```bash
kubectl ate list workers # or inspect Valkey directly
# Worker has Assignment=my-actor but actor is STATUS_SUSPENDED
```
4. Do NOT retry ResumeActor. Observe the worker is never returned to the free pool.

---

## Expected Behavior

A background goroutine (or a startup scan) should detect workers with `Assignment` set
pointing at an actor that is still `STATUS_SUSPENDED`/`STATUS_PAUSED`, and clear the
assignment automatically.

---

## Suggested Fix

In `reconcileOrphanedWorkers` (already runs at startup), also scan for workers where
`Assignment != nil` but the referenced actor has status `SUSPENDED` or `PAUSED`. Clear
the assignment. This closes the window without needing a two-phase commit.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。