agent-substrate / agent-substrate/substrate
[P3] Health checker has no hysteresis — single transient probe failure flips status
- Dominant language
- Go
- Stars
- 1.8k
- Forks
- 316
- Avg merge
- 2d 43m
- Merged PRs (30d)
- 287
Description
**Severity:** P3 (spurious alerts from transient network blips)
**Component:** Network Plane — `cmd/atenet/internal/router/health.go`
**Audit ID:** NET-8
---
## Summary
`updateComponentHealth()` sets `health.Healthy = healthy` directly from each probe
result with no consecutive-failure threshold or smoothing. A single 500ms network blip
to Envoy's admin port (`:9901`) or the Kubernetes API causes `/statusz` to report
unhealthy for one polling interval (default 1s). While this does not affect traffic
routing (health check is observability-only), monitoring systems built on `/statusz`
will page on-call for a 1-second blip.
---
## Root Cause
**File:** `cmd/atenet/internal/router/health.go` lines 147–157
```go
func (h *HealthState) updateComponentHealth(name string, healthy bool) {
h.mu.Lock()
defer h.mu.Unlock()
h.components[name] = ComponentHealth{
Healthy: healthy, // Direct assignment — no threshold
// ...
}
}
```
---
## Steps to Reproduce
```bash
# Drop Envoy admin traffic for 600ms (just over one health check interval)
kubectl exec -n ate-system -- \
iptables -I OUTPUT -p tcp --dport 9901 -j DROP
# Poll /statusz from another terminal
watch -n 0.5 "curl -s http://localhost:8080/statusz | jq '.components.envoy.healthy'"
# Flips to false for one cycle
kubectl exec -n ate-system -- \
iptables -D OUTPUT -p tcp --dport 9901 -j DROP
# Recovers after one successful probe
```
---
## Suggested Fix
Track consecutive failure count per component. Only flip to unhealthy after N
consecutive failures (e.g., N=3):
```go
type ComponentHealth struct {
Healthy bool
consecutiveFails int
// ...
}
func (h *HealthState) updateComponentHealth(name string, healthy bool) {
h.mu.Lock()
defer h.mu.Unlock()
c := h.components[name]
if healthy {
c.consecutiveFails = 0
c.Healthy = true
} else {
c.consecutiveFails++
if c.consecutiveFails >= 3 {
c.Healthy = false
}
}
h.components[name] = c
}
```
Contributor guide
Assessment
This issue has not been assessed yet.