agent-substrate / agent-substrate/substrate
[P2] atenet controller reconcile short-circuits on Kubernetes API error — Envoy Deployment cannot self-heal
- Vorherrschende Sprache
- Go
- Sterne
- 1.8k
- Forks
- 316
- Ø Merge
- 2 T. 43 Min.
- Gemergte PRs (30 T.)
- 287
Beschreibung
**Severity:** P2 (Envoy Deployment drift not corrected during K8s API flap)
**Component:** Network Plane — `cmd/atenet/internal/router/controller.go`
**Audit ID:** NET-5
---
## Summary
The atenet router's `reconcile()` function calls `readyTemplates()` first and returns
early if it fails. `readyTemplates()` performs a K8s API `List` across all namespaces —
an expensive, frequently-failing call during API server stress. The XDS snapshot and
the Envoy Deployment reconciliation (`envoyRunner.reconcile()`) do not depend on
template data, but both are gated behind this check. A Kubernetes API flap prevents
Envoy Deployment self-healing: if the Deployment is manually scaled to 0 during the
flap, it stays at 0 until the API recovers.
---
## Root Cause
**File:** `cmd/atenet/internal/router/controller.go` lines 88–99
```go
func (c *Controller) reconcile(ctx context.Context) error {
templates, err := c.atStore.readyTemplates(ctx)
if err != nil {
c.logger.Error("Failed to get ActorTemplates", "err", err)
return err // ← short-circuit: UpdateSnapshot and envoyRunner.reconcile NEVER RUN
}
if err := c.xdsServer.UpdateSnapshot(ctx, templates); err != nil { ... }
if err := c.envoyRunner.reconcile(ctx); err != nil { ... }
return nil
}
```
The XDS snapshot update (`UpdateSnapshot`) uses only `templates` for route rules, but
an empty template list is a valid (safe) input — it just means no routes are advertised.
`envoyRunner.reconcile()` manages the Envoy Deployment and Service and does not use
`templates` at all.
---
## Steps to Reproduce
1. Simulate a Kubernetes API server flap:
```bash
# On kind: temporarily stop the API server container
docker pause
sleep 30 # API unreachable for 30s
docker unpause
```
2. During the flap, manually scale the Envoy Deployment to 0:
```bash
# Use raw kubectl before docker pause, queue the command
kubectl scale deployment envoy -n ate-system --replicas=0
```
3. After the API recovers, observe:
```bash
kubectl get deployment envoy -n ate-system
# READY: 0/0 ← not recovered to 1/1
```
4. Check controller logs — `reconcile()` was returning early on every cycle during
the flap and never called `envoyRunner.reconcile()` after the API recovered. The
next successful reconcile (which includes readyTemplates) does fix it, but there's
a window where Envoy is down and all actor requests fail.
---
## Expected Behavior
`UpdateSnapshot` and `envoyRunner.reconcile()` should not be gated on `readyTemplates`.
An empty template list is a safe input. A K8s API flap should never prevent Envoy
Deployment self-healing.
---
## Suggested Fix
Split `reconcile` into two independent paths:
```go
func (c *Controller) reconcile(ctx context.Context) error {
// Always runs — does not depend on templates
if err := c.envoyRunner.reconcile(ctx); err != nil {
c.logger.Error("Failed to reconcile Envoy", "err", err)
}
// Best-effort — failure does not block Envoy reconciliation
templates, err := c.atStore.readyTemplates(ctx)
if err != nil {
c.logger.Warn("Failed to get ActorTemplates — XDS snapshot not updated", "err", err)
return err
}
return c.xdsServer.UpdateSnapshot(ctx, templates)
}
```
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.