agent-substrate / agent-substrate/substrate

[P1] atenet-router and dns run as single replicas with no PDB — every restart is a guaranteed full data-plane outage

Abierto
#645 0 comentarios 0 reacciones 0 asignados Ver en GitHub
area/network kind/bug prio/P1
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

> Dedup note: verified against all open+closed issues 2026-07-30 — no issue
> or PR files the data-plane SPOF. #181 ("Define substrate HA story") is
> scoped entirely to ate-apiserver + its database and never mentions the
> router or dns. PR #538 already did HA for ate-api-server (2 replicas +
> round-robin), so a precedent exists. #617/#612/#614 are related but
> distinct (see below).

## Summary

All actor traffic flows through one `atenet-router` pod and all actor name
resolution through one `dns` pod:

- `manifests/ate-install/atenet-router.yaml:110` — `replicas: 1`, no PDB,
no topologySpreadConstraints.
- `manifests/ate-install/atenet-dns.yaml:82` — `replicas: 1`, same.
- The router-managed Envoy Deployment is additionally hardcoded to one
replica in code: `cmd/atenet/internal/router/envoyrunner.go:140`
(`replicas := int32(1)`) — so this is a code change, not just a manifest
bump.
- The only PDB in the repo belongs to ate-api-server
(`manifests/ate-install/ate-api-server.yaml:220`).

Any voluntary or involuntary disruption — deploy, node upgrade, eviction,
OOM — takes the entire data plane down for its duration.

## Evidence (live, GKE substrate-poc, 2026-07-30)

Measured blast radius with an in-cluster probe (5 req/s against a RUNNING
actor) while force-deleting the router pod:

- **~5.5 s of hard connection failures** (curl code `000` — *not* graceful
503s) in the best case: warm image, instant reschedule.
- During the 09:53Z GKE node-pool event the same failure mode produced
**6,880 `dial tcp` errors and 80 EOFs** in the overnight soak — the full
pod-reschedule + image-pull window.

## Impact

- Guaranteed user-visible outage on every router deploy or node drain;
failures surface as raw connection errors, which HTTP clients cannot
safely retry for non-idempotent requests (POST).
- Defeats the "retries heal everything" design story: there is nothing to
retry against while the single router is down.

## Related but distinct (do not fold into this issue)

- #617 — xDS `versionCount` resets on router restart (a *config-staleness*
bug triggered by restart; also a hint that multi-replica xDS needs design
attention, since each replica would keep an independent version counter).
- #612 — atenet controller reconcile short-circuit prevents Envoy
Deployment self-heal.
- #614 — lists "component crash + recovery (… atenet-router)" as an
untested scenario; test-coverage gap, not the SPOF itself.
- #181 — HA umbrella; body covers only ate-apiserver and its store.

## Suggested fix

1. Follow the PR #538 pattern (which HA'd ate-api-server): ≥2 router
replicas behind the Service, PDB `maxUnavailable: 1`, topology spread
across nodes; same for dns. Make `envoyrunner.go`'s replica count
configurable.
2. Resolve the multi-replica xDS question first (#617's version counter
per-replica); if multi-replica is not yet safe, document the SPOF in the
operations runbook (#581) as an interim known limitation.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.