agent-substrate / agent-substrate/substrate

[P2] Transient atelet unavailability defaults to codes.Internal — outside the documented retryable set; strands actor in SUSPENDING and surfaces as http/500

Đang mở
#646 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/api-machinery area/network kind/bug prio/P2
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

> Dedup note: verified against all open+closed issues 2026-07-30 — nothing
> files the Internal-misclassification of transient infra conditions or the
> SUSPENDING-strand scenario. Closest relatives: #605 (inverse
> classification bug: terminal ENOSPC treated as retriable), #50 (same
> SUSPENDING symptom, different root cause — runsc exit 128, retry does NOT
> converge there), #554 (reentrancy umbrella), #600 (precedent for the
> "Terminating pod has no IP" window, worker-pod side).

## Summary

When the atelet DaemonSet pod on an actor's node is being replaced (routine
during node upgrades), SuspendActor fails with:

```
rpc error: code = Internal desc = internal server error: workflow failed at
step CallAteletSuspend: while getting atelet conn for worker pod: selected
atelet "ate-system/atelet-w78v7" has no assigned IPs
```

Mechanism (verified on main):
- `cmd/ateapi/internal/controlapi/dialer.go:112` produces the error as a
plain `fmt.Errorf` — no gRPC code attached.
- `workflow_suspend.go:140` wraps it with `%w` — still no code.
- `internal/ateinterceptors/ateinterceptors.go:72` then defaults every
non-status error to `codes.Internal "internal server error"`.

Why the code matters — this is not a style nit; the system documents
contracts around specific codes:
1. `docs/request-parking.md`: the router parks/retries only
`FailedPrecondition`, `Aborted`, `Unavailable`; everything else is
fail-fast. A self-healing ~10 s condition lands outside the retryable
set.
2. Router HTTP mapping (PR #68): `Unavailable`/`FailedPrecondition` → 503,
"anything else" → **500**. So Internal-classified churn errors surface
to HTTP callers as 500s — which matches the **1,840 unexplained
`http/500`** responses in the 2026-07-30 overnight soak (during
ateapi/atelet chaos kills).

## Live reproduction (GKE substrate-poc, 2026-07-30)

`kubectl delete pod -n ate-system --grace-period=0`,
then SuspendActor on the RUNNING actor hosted there ~0.5 s later:
- SuspendActor returned the `Internal` error above.
- Actor left in `STATUS_SUSPENDING`. In-cluster HTTP probe went from steady
200s to hanging until its 5 s client timeout — consistent with requests
being parked on the resume path's `FailedPrecondition` (SUSPENDING is not
resumable) with nothing repairing the state, rather than a fast 503.
- A retried SuspendActor after the replacement atelet got its IP converged
cleanly (SUSPENDED → auto-resume on next request). Forward recovery
works; the defect is purely that the error classification tells clients
*not* to retry.

## Suggested fix

- In the dialer / workflow step, classify "no atelet conn / atelet has no
assigned IPs / atelet not found" as `codes.Unavailable`.
- Audit other plain `fmt.Errorf` paths in workflow steps that hit the
`ateinterceptors.go:72` Internal default for infrastructure-transient
conditions (the #605 ENOSPC bug is the same audit from the opposite
direction: both directions of misclassification exist).
- Router: ensure upstream `Unavailable` during wake maps to 503 +
`Retry-After`, never a park-until-timeout hang.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.