envoyproxy / envoyproxy/gateway
Improve the watchable control-plane metrics: watchable_depth is always 0, and subscribe duration buckets are too coarse below 10s
- Dominant language
- Go
- Stars
- 3k
- Forks
- 864
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 140
Description
Two follow-ups from #9773, both about the `watchable_*` control-plane metrics being hard to read. They are independent and can be done separately.
- [ ] **1. `watchable_depth` is always 0, and the dashboard graphs it**
`watchable_depth` is recorded as `len(snapshots)` (`internal/message/watchutil.go:135`), where `snapshots` is the channel returned by `watchable.Map.Subscribe()`. That channel is unbuffered — `downstream := make(chan Snapshot[K, V])` in the watchable library — so the gauge can only ever be 0.
This is not just dead weight. The shipped Grafana dashboard graphs it, so operators see a flat zero and can reasonably read it as "no backlog" while the control plane is in fact seconds behind. That misreading happened during a real incident investigation.
Removing the metric is not straightforward, because the dashboard also uses it as a template variable source, so the `$Runner` and `$Namespace` dropdowns would break and take every filtered panel with them:
```
charts/gateway-addons-helm/dashboards/envoy-gateway-global.json
277 "expr": "sum by(runner) (watchable_depth{runner=~\"$Runner\", namespace=\"$Namespace\"})"
3046 "definition": "label_values(watchable_depth,namespace)"
3078 "definition": "label_values(watchable_depth,runner)"
```
Suggested order of work:
- Repoint the depth panel at something meaningful. `watchable_debounce_pending` (added in #9773) reports how many updates were merged per flush, but only when debouncing is enabled, so a panel that works in both modes may need a different signal.
- Move the two `label_values` queries onto a metric that is always present with the same `runner`/`namespace` labels — `watchable_subscribe_total` would do.
- Only then decide whether to deprecate and remove the gauge, or redefine what it measures. Either is a breaking change for anyone with their own dashboards or alerts, so it wants a deprecation note.
- [x] **2. `watchable_subscribe_duration_seconds` has poor resolution below 10s**
The buckets are `{0.001, 0.01, 0.1, 1, 5, 10, 30, 60, 120}` (`internal/message/metrics.go:31`). #9773 appended `30, 60, 120` so the tail is no longer hidden, but deliberately did not touch the existing boundaries, since removing them is breaking.
The gaps that remain are `0.1 → 1` (10x) and `1 → 5` (5x). A control plane whose translations take ~2s puts every observation in `(1, 5]`, and because `histogram_quantile` interpolates within a bucket, p50/p95/p99 all come back somewhere in 1–5s with no discriminating power — the same numbers whether things are healthy or steadily degrading, until they cross 5s. A sub-second control plane has the same problem in `(0.1, 1]`.
This repo already has better-spaced buckets for the same kind of measurement, from the k8s rest client (`internal/metrics/restclient/metrics.go:24`):
```go
[]float64{0.005, 0.025, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 15.0, 30.0, 60.0}
```
Adopting that spacing, extended to 120s, would give roughly 2–2.5x steps throughout. It removes the `0.001`, `1` and `5` boundaries, so anything referencing those `le` values breaks — hence a separate change with a breaking-change note.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with internal/message/watchutil.go:135 and internal/message/metrics.go:31, then inspect charts/gateway-addons-helm/dashboards/envoy-gateway-global.json and the bucket spacing in internal/metrics/restclient/metrics.go:24. Treat the two checklist items separately, preserving the dashboard dropdowns while choosing a meaningful depth signal and revising the duration buckets. Done means the dashboard no longer presents a permanently zero depth, the label queries still work, and the bucket or metric compatibility changes have a deprecation note.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grafana
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100