agent-substrate / agent-substrate/substrate

[P3] Health checker has no hysteresis — single transient probe failure flips status

Đang mở
#616 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/network kind/bug prio/P2
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

**Severity:** P3 (spurious alerts from transient network blips)
**Component:** Network Plane — `cmd/atenet/internal/router/health.go`
**Audit ID:** NET-8

---

## Summary

`updateComponentHealth()` sets `health.Healthy = healthy` directly from each probe
result with no consecutive-failure threshold or smoothing. A single 500ms network blip
to Envoy's admin port (`:9901`) or the Kubernetes API causes `/statusz` to report
unhealthy for one polling interval (default 1s). While this does not affect traffic
routing (health check is observability-only), monitoring systems built on `/statusz`
will page on-call for a 1-second blip.

---

## Root Cause

**File:** `cmd/atenet/internal/router/health.go` lines 147–157

```go
func (h *HealthState) updateComponentHealth(name string, healthy bool) {
h.mu.Lock()
defer h.mu.Unlock()
h.components[name] = ComponentHealth{
Healthy: healthy, // Direct assignment — no threshold
// ...
}
}
```

---

## Steps to Reproduce

```bash
# Drop Envoy admin traffic for 600ms (just over one health check interval)
kubectl exec -n ate-system -- \
iptables -I OUTPUT -p tcp --dport 9901 -j DROP

# Poll /statusz from another terminal
watch -n 0.5 "curl -s http://localhost:8080/statusz | jq '.components.envoy.healthy'"
# Flips to false for one cycle

kubectl exec -n ate-system -- \
iptables -D OUTPUT -p tcp --dport 9901 -j DROP
# Recovers after one successful probe
```

---

## Suggested Fix

Track consecutive failure count per component. Only flip to unhealthy after N
consecutive failures (e.g., N=3):

```go
type ComponentHealth struct {
Healthy bool
consecutiveFails int
// ...
}

func (h *HealthState) updateComponentHealth(name string, healthy bool) {
h.mu.Lock()
defer h.mu.Unlock()
c := h.components[name]
if healthy {
c.consecutiveFails = 0
c.Healthy = true
} else {
c.consecutiveFails++
if c.consecutiveFails >= 3 {
c.Healthy = false
}
}
h.components[name] = c
}
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.