agent-substrate / agent-substrate/substrate

[P2] Transient atelet unavailability defaults to codes.Internal — outside the documented retryable set; strands actor in SUSPENDING and surfaces as http/500

Offen
#646 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
area/api-machinery area/network kind/bug prio/P2
Vorherrschende Sprache
Go
Sterne
1.8k
Forks
316
Ø Merge
2 T. 43 Min.
Gemergte PRs (30 T.)
287

Beschreibung

> Dedup note: verified against all open+closed issues 2026-07-30 — nothing
> files the Internal-misclassification of transient infra conditions or the
> SUSPENDING-strand scenario. Closest relatives: #605 (inverse
> classification bug: terminal ENOSPC treated as retriable), #50 (same
> SUSPENDING symptom, different root cause — runsc exit 128, retry does NOT
> converge there), #554 (reentrancy umbrella), #600 (precedent for the
> "Terminating pod has no IP" window, worker-pod side).

## Summary

When the atelet DaemonSet pod on an actor's node is being replaced (routine
during node upgrades), SuspendActor fails with:

```
rpc error: code = Internal desc = internal server error: workflow failed at
step CallAteletSuspend: while getting atelet conn for worker pod: selected
atelet "ate-system/atelet-w78v7" has no assigned IPs
```

Mechanism (verified on main):
- `cmd/ateapi/internal/controlapi/dialer.go:112` produces the error as a
plain `fmt.Errorf` — no gRPC code attached.
- `workflow_suspend.go:140` wraps it with `%w` — still no code.
- `internal/ateinterceptors/ateinterceptors.go:72` then defaults every
non-status error to `codes.Internal "internal server error"`.

Why the code matters — this is not a style nit; the system documents
contracts around specific codes:
1. `docs/request-parking.md`: the router parks/retries only
`FailedPrecondition`, `Aborted`, `Unavailable`; everything else is
fail-fast. A self-healing ~10 s condition lands outside the retryable
set.
2. Router HTTP mapping (PR #68): `Unavailable`/`FailedPrecondition` → 503,
"anything else" → **500**. So Internal-classified churn errors surface
to HTTP callers as 500s — which matches the **1,840 unexplained
`http/500`** responses in the 2026-07-30 overnight soak (during
ateapi/atelet chaos kills).

## Live reproduction (GKE substrate-poc, 2026-07-30)

`kubectl delete pod -n ate-system --grace-period=0`,
then SuspendActor on the RUNNING actor hosted there ~0.5 s later:
- SuspendActor returned the `Internal` error above.
- Actor left in `STATUS_SUSPENDING`. In-cluster HTTP probe went from steady
200s to hanging until its 5 s client timeout — consistent with requests
being parked on the resume path's `FailedPrecondition` (SUSPENDING is not
resumable) with nothing repairing the state, rather than a fast 503.
- A retried SuspendActor after the replacement atelet got its IP converged
cleanly (SUSPENDED → auto-resume on next request). Forward recovery
works; the defect is purely that the error classification tells clients
*not* to retry.

## Suggested fix

- In the dialer / workflow step, classify "no atelet conn / atelet has no
assigned IPs / atelet not found" as `codes.Unavailable`.
- Audit other plain `fmt.Errorf` paths in workflow steps that hit the
`ateinterceptors.go:72` Internal default for infrastructure-transient
conditions (the #605 ENOSPC bug is the same audit from the opposite
direction: both directions of misclassification exist).
- Router: ensure upstream `Unavailable` during wake maps to 503 +
`Retry-After`, never a park-until-timeout hang.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.