[P1] Actor lock TTL (30s) exceeds router retry budget (15s) — ateapi crash causes guaranteed 15–30s user-visible outage

**Severity:** P1 (guaranteed user-visible failure window after any ateapi crash)
**Component:** Control Plane + Network Plane — `ateredis.go` + `resumer.go`
**Audit ID:** CP-4, NET-3, NET-6
**Maps to suspect:** S12

---

## Summary

When ateapi crashes mid-workflow, the actor's advisory lock (stored in Valkey with a
30-second TTL) remains held by the dead process. During the 30-second window before
the TTL expires, every `ResumeActor` attempt returns `codes.Aborted` ("another
operation is in progress"). The router's fail-fast retry budget is 15 seconds, and
Envoy's ext_proc timeout is 5 seconds — both shorter than the lock TTL. Every client
request during the 30-second window fails. Additionally, late-joining requests inherit
the remaining budget of the in-flight singleflight, potentially failing immediately if
they join near expiry.

---

## Impact

- Up to 30 seconds of hard actor unavailability after every ateapi pod crash or rolling
restart.
- All client requests during this window fail with opaque 503/504 errors.
- The error message does not distinguish "ateapi crashed" from "no capacity" — on-call
cannot immediately diagnose the cause.
- With a rolling restart of 3 ateapi replicas and a 10s restart interval, the total
outage window can be 30s per pod × 3 = 90s for actors being actively used during
the rollout.

---

## Root Cause

**File:** `cmd/ateapi/internal/store/ateredis/ateredis.go` line 805:
```go
defaultLockTTL = 30 * time.Second
```

**File:** `cmd/atenet/internal/router/resumer.go` line 36:
```go
failFastResumeBudget = 15 * time.Second
```

**File:** `cmd/atenet/internal/router/xds.go` line 83:
```go
defaultExtProcMessageTimeout = 5 * time.Second
```

The lock TTL goroutine that renews the lock dies with the ateapi process. The lock
key remains in Valkey until the TTL expires. The gap:

```
Lock TTL (30s) >> Envoy ext_proc timeout (5s) ≈ fail-fast budget (15s, but Envoy
cuts it at 5s) >> effective user retry budget (~5s)
```

Additionally, the singleflight in `resumer.go` uses a background context with a fixed
budget created when the **first** caller triggers the flight (lines 142–196). All
subsequent callers share the remaining budget. A caller joining at second 14 of a 15s
flight gets only 1 second — and under the S12 lock scenario this means:

- T=0: first flight starts, retrying `Aborted` every ~100ms for 15s.
- T=14.9: second request joins flight, gets 100ms budget → immediately exhausted.
- T=15: first flight exhausts, returns 503.
- T=15.1: third request starts a new flight. Lock still has ~15s remaining. Same cycle.
- T=30: lock expires. Next request succeeds.

---

## Steps to Reproduce

1. Have an actor in `STATUS_SUSPENDED`.
2. Send `ResumeActor` and kill ateapi immediately after the actor lock is acquired
(watch ateapi logs for the workflow start):
```bash
kubectl ate resume actor my-actor -a demo &
sleep 0.5
kubectl delete pod -n ate-system -l app=ate-api-server --wait=false
```
3. Send requests through the router immediately:
```bash
for i in $(seq 1 20); do
time curl -w "%{http_code}" -H "Host: my-actor.demo.actors.resources.substrate.ate.dev" \
http://localhost:8000
sleep 1
done
```
4. Observe:
- Requests fail with 503/504 for up to 30 seconds.
- After ~30 seconds (lock TTL expires), requests start succeeding again.
5. Check router logs to confirm `Aborted` codes from ateapi during the window:
```bash
kubectl logs -n ate-system deployment/atenet-router | grep "Aborted\|another operation"
```

---

## Expected Behavior

An ateapi crash should not create a user-visible failure window longer than the normal
request timeout (~1 second). Recovery options:

1. **Short lock TTL**: reduce `defaultLockTTL` to ≤ 10s (within one router retry cycle).
2. **Lock release on startup**: on ateapi startup, scan and release any locks held by
a pod UID that no longer exists.
3. **Extend router budget**: raise `failFastResumeBudget` > 30s, but this increases
latency for truly-saturated pools.

---

## Actual Behavior

Lock TTL (30s) > router budget (15s) > Envoy timeout (5s). Every request fails for
up to 30s after ateapi crash. The failure is repeated and consistent, not a one-time
blip.

---

## Suggested Fix

**Immediate (reduce TTL):** Reduce `defaultLockTTL` from 30s to 10s. This cuts the
maximum outage window to 10s, which is within the `failFastResumeBudget` and gives the
router a chance to outlast the lock.

**Proper fix (startup cleanup):** On ateapi startup, after acquiring the Kubernetes
leader lease, scan all actor locks in Valkey. For each lock, check if the holder pod
UID still exists in the cluster. If not, delete the lock key immediately. This reduces
the outage window to the ateapi restart time (~5–15s on GKE), not the lock TTL.

**Also fix:** Extend the singleflight so each individual caller gets its own deadline
rather than sharing the flight's remaining budget. This prevents late joiners from
seeing immediate exhaustion under the S12 scenario.