agent-substrate / agent-substrate/substrate

[P3] Health checker has no hysteresis — single transient probe failure flips status

Abierto
#616 0 comentarios 0 reacciones 0 asignados Ver en GitHub
area/network kind/bug prio/P2
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

**Severity:** P3 (spurious alerts from transient network blips)
**Component:** Network Plane — `cmd/atenet/internal/router/health.go`
**Audit ID:** NET-8

---

## Summary

`updateComponentHealth()` sets `health.Healthy = healthy` directly from each probe
result with no consecutive-failure threshold or smoothing. A single 500ms network blip
to Envoy's admin port (`:9901`) or the Kubernetes API causes `/statusz` to report
unhealthy for one polling interval (default 1s). While this does not affect traffic
routing (health check is observability-only), monitoring systems built on `/statusz`
will page on-call for a 1-second blip.

---

## Root Cause

**File:** `cmd/atenet/internal/router/health.go` lines 147–157

```go
func (h *HealthState) updateComponentHealth(name string, healthy bool) {
h.mu.Lock()
defer h.mu.Unlock()
h.components[name] = ComponentHealth{
Healthy: healthy, // Direct assignment — no threshold
// ...
}
}
```

---

## Steps to Reproduce

```bash
# Drop Envoy admin traffic for 600ms (just over one health check interval)
kubectl exec -n ate-system -- \
iptables -I OUTPUT -p tcp --dport 9901 -j DROP

# Poll /statusz from another terminal
watch -n 0.5 "curl -s http://localhost:8080/statusz | jq '.components.envoy.healthy'"
# Flips to false for one cycle

kubectl exec -n ate-system -- \
iptables -D OUTPUT -p tcp --dport 9901 -j DROP
# Recovers after one successful probe
```

---

## Suggested Fix

Track consecutive failure count per component. Only flip to unhealthy after N
consecutive failures (e.g., N=3):

```go
type ComponentHealth struct {
Healthy bool
consecutiveFails int
// ...
}

func (h *HealthState) updateComponentHealth(name string, healthy bool) {
h.mu.Lock()
defer h.mu.Unlock()
c := h.components[name]
if healthy {
c.consecutiveFails = 0
c.Healthy = true
} else {
c.consecutiveFails++
if c.consecutiveFails >= 3 {
c.Healthy = false
}
}
h.components[name] = c
}
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.