agent-substrate / agent-substrate/substrate

[P2] Worker slot leaked when ateapi crashes between UpdateWorker and UpdateActor

Đang mở
#607 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/api-machinery area/scheduling kind/bug prio/P2
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

**Severity:** P2 (self-heals on user retry or pod death; reduces pool capacity in interim)
**Component:** Control Plane — `cmd/ateapi/internal/controlapi/workflow_resume.go`
**Audit ID:** CP-1
**Maps to suspect:** S1

---

## Summary

`AssignWorkerStep.Execute` claims a worker by calling `store.UpdateWorker` (sets
`Assignment`) before calling `store.UpdateActor` (sets `STATUS_RESUMING`). If ateapi
crashes between these two writes, the worker record has an `Assignment` pointing at the
actor but the actor is still `STATUS_SUSPENDED`. No background reclaimer detects this
stale claim. Pool capacity is reduced by one until the user retries `ResumeActor` (which
triggers the recovery loop) or the worker pod is deleted.

---

## Impact

- Worker pool capacity silently reduced by 1 per occurrence.
- In a busy cluster with frequent user abandonment of actors, this drains the pool
over time.
- No monitoring metric or alert for "worker claimed but actor not resuming."

---

## Root Cause

**File:** `cmd/ateapi/internal/controlapi/workflow_resume.go` lines 218–250

```go
// Line 218: claims the worker
if err := store.UpdateWorker(..., Assignment: actorUID); err != nil { ... }
// --- CRASH WINDOW ---
// Line 229: marks actor RESUMING
if err := store.UpdateActor(..., Status: STATUS_RESUMING); err != nil { ... }
```

The recovery loop at lines 165–193 fires only at the start of a new `ResumeActor` RPC,
scanning for a worker whose `Assignment == actorUID` when the actor is still SUSPENDED.
If the user never retries, the claim persists until pod death.

---

## Steps to Reproduce

1. Have a `STATUS_SUSPENDED` actor and a free worker pool.
2. Call `ResumeActor`, kill ateapi after `UpdateWorker` returns but before `UpdateActor`:
```bash
kubectl ate resume actor my-actor -a demo &
sleep 0.3 # enough for UpdateWorker to complete
kubectl delete pod -n ate-system -l app=ate-api-server
```
3. After ateapi restarts, list workers:
```bash
kubectl ate list workers # or inspect Valkey directly
# Worker has Assignment=my-actor but actor is STATUS_SUSPENDED
```
4. Do NOT retry ResumeActor. Observe the worker is never returned to the free pool.

---

## Expected Behavior

A background goroutine (or a startup scan) should detect workers with `Assignment` set
pointing at an actor that is still `STATUS_SUSPENDED`/`STATUS_PAUSED`, and clear the
assignment automatically.

---

## Suggested Fix

In `reconcileOrphanedWorkers` (already runs at startup), also scan for workers where
`Assignment != nil` but the referenced actor has status `SUSPENDED` or `PAUSED`. Clear
the assignment. This closes the window without needing a two-phase commit.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.