envoyproxy / envoyproxy/gateway

Improve the watchable control-plane metrics: watchable_depth is always 0, and subscribe duration buckets are too coarse below 10s

Open
#9,776 1 comment 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

Two follow-ups from #9773, both about the `watchable_*` control-plane metrics being hard to read. They are independent and can be done separately.

- [ ] **1. `watchable_depth` is always 0, and the dashboard graphs it**

`watchable_depth` is recorded as `len(snapshots)` (`internal/message/watchutil.go:135`), where `snapshots` is the channel returned by `watchable.Map.Subscribe()`. That channel is unbuffered — `downstream := make(chan Snapshot[K, V])` in the watchable library — so the gauge can only ever be 0.

This is not just dead weight. The shipped Grafana dashboard graphs it, so operators see a flat zero and can reasonably read it as "no backlog" while the control plane is in fact seconds behind. That misreading happened during a real incident investigation.

Removing the metric is not straightforward, because the dashboard also uses it as a template variable source, so the `$Runner` and `$Namespace` dropdowns would break and take every filtered panel with them:

```
charts/gateway-addons-helm/dashboards/envoy-gateway-global.json
277 "expr": "sum by(runner) (watchable_depth{runner=~\"$Runner\", namespace=\"$Namespace\"})"
3046 "definition": "label_values(watchable_depth,namespace)"
3078 "definition": "label_values(watchable_depth,runner)"
```

Suggested order of work:

- Repoint the depth panel at something meaningful. `watchable_debounce_pending` (added in #9773) reports how many updates were merged per flush, but only when debouncing is enabled, so a panel that works in both modes may need a different signal.
- Move the two `label_values` queries onto a metric that is always present with the same `runner`/`namespace` labels — `watchable_subscribe_total` would do.
- Only then decide whether to deprecate and remove the gauge, or redefine what it measures. Either is a breaking change for anyone with their own dashboards or alerts, so it wants a deprecation note.

- [x] **2. `watchable_subscribe_duration_seconds` has poor resolution below 10s**

The buckets are `{0.001, 0.01, 0.1, 1, 5, 10, 30, 60, 120}` (`internal/message/metrics.go:31`). #9773 appended `30, 60, 120` so the tail is no longer hidden, but deliberately did not touch the existing boundaries, since removing them is breaking.

The gaps that remain are `0.1 → 1` (10x) and `1 → 5` (5x). A control plane whose translations take ~2s puts every observation in `(1, 5]`, and because `histogram_quantile` interpolates within a bucket, p50/p95/p99 all come back somewhere in 1–5s with no discriminating power — the same numbers whether things are healthy or steadily degrading, until they cross 5s. A sub-second control plane has the same problem in `(0.1, 1]`.

This repo already has better-spaced buckets for the same kind of measurement, from the k8s rest client (`internal/metrics/restclient/metrics.go:24`):

```go
[]float64{0.005, 0.025, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0, 15.0, 30.0, 60.0}
```

Adopting that spacing, extended to 120s, would give roughly 2–2.5x steps throughout. It removes the `0.001`, `1` and `5` boundaries, so anything referencing those `le` values breaks — hence a separate change with a breaking-change note.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with internal/message/watchutil.go:135 and internal/message/metrics.go:31, then inspect charts/gateway-addons-helm/dashboards/envoy-gateway-global.json and the bucket spacing in internal/metrics/restclient/metrics.go:24. Treat the two checklist items separately, preserving the dashboard dropdowns while choosing a meaningful depth signal and revising the duration buckets. Done means the dashboard no longer presents a permanently zero depth, the label queries still work, and the bucket or metric compatibility changes have a deprecation note.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grafana
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.