pingcap / pingcap/tidb

metrics: show Disconnection Count as a rate, split out undetermined as a separate cumulative counter

Open
#70,336 6 comments 0 reactions 0 assignees View on GitHub
contribution needs-cherry-pick-release-8.5 type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

Currently, the `TiDB` dashboard's `Server` row `Disconnection Count` panel plots the raw cumulative counter directly:

```
sum(tidb_server_disconnection_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance"}) by (instance, result)
```

Panel definition: `pkg/metrics/grafana/tidb.json` (`Server` row, `Disconnection Count` panel, id 205). The same panel also exists in `pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json` and `pkg/metrics/nextgengrafana/tidb_worker.json`.

In a long-running production cluster this counter accumulates to very large numbers, so the raw value has little practical use for spotting current connect/disconnect activity.

### Proposed change

1. Switch the `ok`/`error` result series to a rate view with `irate(...[30s])`. This makes the panel show disconnect frequency directly, and — since disconnects roughly track new-connection creation over time — lets users eyeball whether the workload looks like long-lived or short-lived connections.
2. Keep `result="undetermined"` as its own series, still plotted as a cumulative total, on the panel's right Y axis. Any nonzero `undetermined` count means a transaction was once left in a state TiDB could not automatically resolve and needs manual investigation, so this one should stay an absolute "has this ever happened" counter rather than being folded into a rate.

Draft query changes (prototyped and manually verified against a local `tiup playground` Grafana instance):

```
# ok/error rate, left axis
sum(irate(tidb_server_disconnection_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", result!="undetermined"}[30s])) by (instance, result)

# undetermined cumulative total, right axis (via a seriesOverride routing it to yaxis 2, unstacked)
sum(tidb_server_disconnection_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", result="undetermined"}) by (instance)
```

Affected dashboard files: `pkg/metrics/grafana/tidb.json`, `pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json`, `pkg/metrics/nextgengrafana/tidb_worker.json`.

/label component/observability

Contributor guide

Open the contributing guide

Research direction

Update panel 205 in pkg/metrics/grafana/tidb.json and its counterparts in pkg/metrics/nextgengrafana/tidb_with_keyspace_name.json and pkg/metrics/nextgengrafana/tidb_worker.json. Start by comparing the existing panel queries and seriesOverride settings; use the proposed irate query for ok/error and keep undetermined cumulative on the right axis, unstacked. Verify the three dashboards in a local tiup playground Grafana instance.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.