agent-substrate / agent-substrate/substrate
[P1] atenet-router and dns run as single replicas with no PDB — every restart is a guaranteed full data-plane outage
- Vorherrschende Sprache
- Go
- Sterne
- 1.8k
- Forks
- 316
- Ø Merge
- 2 T. 43 Min.
- Gemergte PRs (30 T.)
- 287
Beschreibung
> Dedup note: verified against all open+closed issues 2026-07-30 — no issue
> or PR files the data-plane SPOF. #181 ("Define substrate HA story") is
> scoped entirely to ate-apiserver + its database and never mentions the
> router or dns. PR #538 already did HA for ate-api-server (2 replicas +
> round-robin), so a precedent exists. #617/#612/#614 are related but
> distinct (see below).
## Summary
All actor traffic flows through one `atenet-router` pod and all actor name
resolution through one `dns` pod:
- `manifests/ate-install/atenet-router.yaml:110` — `replicas: 1`, no PDB,
no topologySpreadConstraints.
- `manifests/ate-install/atenet-dns.yaml:82` — `replicas: 1`, same.
- The router-managed Envoy Deployment is additionally hardcoded to one
replica in code: `cmd/atenet/internal/router/envoyrunner.go:140`
(`replicas := int32(1)`) — so this is a code change, not just a manifest
bump.
- The only PDB in the repo belongs to ate-api-server
(`manifests/ate-install/ate-api-server.yaml:220`).
Any voluntary or involuntary disruption — deploy, node upgrade, eviction,
OOM — takes the entire data plane down for its duration.
## Evidence (live, GKE substrate-poc, 2026-07-30)
Measured blast radius with an in-cluster probe (5 req/s against a RUNNING
actor) while force-deleting the router pod:
- **~5.5 s of hard connection failures** (curl code `000` — *not* graceful
503s) in the best case: warm image, instant reschedule.
- During the 09:53Z GKE node-pool event the same failure mode produced
**6,880 `dial tcp` errors and 80 EOFs** in the overnight soak — the full
pod-reschedule + image-pull window.
## Impact
- Guaranteed user-visible outage on every router deploy or node drain;
failures surface as raw connection errors, which HTTP clients cannot
safely retry for non-idempotent requests (POST).
- Defeats the "retries heal everything" design story: there is nothing to
retry against while the single router is down.
## Related but distinct (do not fold into this issue)
- #617 — xDS `versionCount` resets on router restart (a *config-staleness*
bug triggered by restart; also a hint that multi-replica xDS needs design
attention, since each replica would keep an independent version counter).
- #612 — atenet controller reconcile short-circuit prevents Envoy
Deployment self-heal.
- #614 — lists "component crash + recovery (… atenet-router)" as an
untested scenario; test-coverage gap, not the SPOF itself.
- #181 — HA umbrella; body covers only ate-apiserver and its store.
## Suggested fix
1. Follow the PR #538 pattern (which HA'd ate-api-server): ≥2 router
replicas behind the Service, PDB `maxUnavailable: 1`, topology spread
across nodes; same for dns. Make `envoyrunner.go`'s replica count
configurable.
2. Resolve the multi-replica xDS question first (#617's version counter
per-replica); if multi-replica is not yet safe, document the SPOF in the
operations runbook (#581) as an interim known limitation.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.