agent-substrate / agent-substrate/substrate

[P2] Worker slot leaked when ateapi crashes between UpdateWorker and UpdateActor

Ouverte
#607 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub
area/api-machinery area/scheduling kind/bug prio/P2
Langage dominant
Go
Étoiles
1.8k
Forks
316
Merge moyen
2 j 43 min
PR mergées (30 j)
287

Description

**Severity:** P2 (self-heals on user retry or pod death; reduces pool capacity in interim)
**Component:** Control Plane — `cmd/ateapi/internal/controlapi/workflow_resume.go`
**Audit ID:** CP-1
**Maps to suspect:** S1

---

## Summary

`AssignWorkerStep.Execute` claims a worker by calling `store.UpdateWorker` (sets
`Assignment`) before calling `store.UpdateActor` (sets `STATUS_RESUMING`). If ateapi
crashes between these two writes, the worker record has an `Assignment` pointing at the
actor but the actor is still `STATUS_SUSPENDED`. No background reclaimer detects this
stale claim. Pool capacity is reduced by one until the user retries `ResumeActor` (which
triggers the recovery loop) or the worker pod is deleted.

---

## Impact

- Worker pool capacity silently reduced by 1 per occurrence.
- In a busy cluster with frequent user abandonment of actors, this drains the pool
over time.
- No monitoring metric or alert for "worker claimed but actor not resuming."

---

## Root Cause

**File:** `cmd/ateapi/internal/controlapi/workflow_resume.go` lines 218–250

```go
// Line 218: claims the worker
if err := store.UpdateWorker(..., Assignment: actorUID); err != nil { ... }
// --- CRASH WINDOW ---
// Line 229: marks actor RESUMING
if err := store.UpdateActor(..., Status: STATUS_RESUMING); err != nil { ... }
```

The recovery loop at lines 165–193 fires only at the start of a new `ResumeActor` RPC,
scanning for a worker whose `Assignment == actorUID` when the actor is still SUSPENDED.
If the user never retries, the claim persists until pod death.

---

## Steps to Reproduce

1. Have a `STATUS_SUSPENDED` actor and a free worker pool.
2. Call `ResumeActor`, kill ateapi after `UpdateWorker` returns but before `UpdateActor`:
```bash
kubectl ate resume actor my-actor -a demo &
sleep 0.3 # enough for UpdateWorker to complete
kubectl delete pod -n ate-system -l app=ate-api-server
```
3. After ateapi restarts, list workers:
```bash
kubectl ate list workers # or inspect Valkey directly
# Worker has Assignment=my-actor but actor is STATUS_SUSPENDED
```
4. Do NOT retry ResumeActor. Observe the worker is never returned to the free pool.

---

## Expected Behavior

A background goroutine (or a startup scan) should detect workers with `Assignment` set
pointing at an actor that is still `STATUS_SUSPENDED`/`STATUS_PAUSED`, and clear the
assignment automatically.

---

## Suggested Fix

In `reconcileOrphanedWorkers` (already runs at startup), also scan for workers where
`Assignment != nil` but the referenced actor has status `SUSPENDED` or `PAUSED`. Clear
the assignment. This closes the window without needing a two-phase commit.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.