metrics: show Disconnection Count as a rate, split out undetermined as a separate cumulative counter
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
Currently, the `TiDB` dashboard's `Server` row `Disconnection Count` panel plots the raw cumulative counter directly:
```
sum(tidb_server_disconnection_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance"}) by (instance, result)
```
Panel definition: `pkg/metrics/grafana/tidb.json` (`Server` row, `Disconnection Count` panel, id 205). The same panel also exists in `pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json` and `pkg/metrics/nextgengrafana/tidb_worker.json`.
In a long-running production cluster this counter accumulates to very large numbers, so the raw value has little practical use for spotting current connect/disconnect activity.
### Proposed change
1. Switch the `ok`/`error` result series to a rate view with `irate(...[30s])`. This makes the panel show disconnect frequency directly, and — since disconnects roughly track new-connection creation over time — lets users eyeball whether the workload looks like long-lived or short-lived connections.
2. Keep `result="undetermined"` as its own series, still plotted as a cumulative total, on the panel's right Y axis. Any nonzero `undetermined` count means a transaction was once left in a state TiDB could not automatically resolve and needs manual investigation, so this one should stay an absolute "has this ever happened" counter rather than being folded into a rate.
Draft query changes (prototyped and manually verified against a local `tiup playground` Grafana instance):
```
# ok/error rate, left axis
sum(irate(tidb_server_disconnection_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", result!="undetermined"}[30s])) by (instance, result)
# undetermined cumulative total, right axis (via a seriesOverride routing it to yaxis 2, unstacked)
sum(tidb_server_disconnection_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", result="undetermined"}) by (instance)
```
Affected dashboard files: `pkg/metrics/grafana/tidb.json`, `pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json`, `pkg/metrics/nextgengrafana/tidb_worker.json`.
/label component/observability
Contributor guide
Research direction
Update panel 205 in pkg/metrics/grafana/tidb.json and its counterparts in pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json and pkg/metrics/nextgengrafana/tidb_worker.json. Start by comparing the existing panel queries and seriesOverride settings; use the proposed irate query for ok/error and keep undetermined cumulative on the right axis, unstacked. Verify the three dashboards in a local tiup playground Grafana instance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100