metrics: connection-lifecycle metrics are missing the resource_group label
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
Connection-lifecycle metrics carry no `resource_group` label, so on a cluster where several applications share TiDB through resource groups it is impossible to attribute connection creation, teardown, idle time or KILL events to a resource group. Query-side metrics already have the label; the connection side does not.
### Current state
Metrics in `pkg/metrics/*.go` that carry `LblResourceGroup` — **identical on `master` and `v8.5.3`**, 12 in total:
```
connections execute_error_total
expensive_total handle_query_duration_seconds
packet_io_bytes prepared_stmts
query_runaway_check query_statement_processed_keys
query_total resource_group_query_total
statement_total transaction_pessimistic_dml_duration_by_attempt
```
The connection-lifecycle metrics do not:
| metric | label set on `master` | defined at |
| --- | --- | --- |
| `tidb_server_disconnection_total` | `{result}` | `pkg/metrics/server.go:142` |
| `tidb_server_handshake_error_total` | none | `pkg/metrics/server.go:239` |
| `tidb_server_event_total` (includes `type="kill"`) | `{type}` | `pkg/metrics/server.go:173` |
| `tidb_server_conn_idle_duration_seconds` | `{in_txn}` | `pkg/metrics/server.go:318` |
| `tidb_server_get_token_duration_seconds` | none | `pkg/metrics/server.go:248` |
| `tidb_executor_phase_duration_seconds` | `{phase, internal}` | `pkg/metrics/executor.go:115` |
`tidb_server_connections` is a gauge and does have the label, so the *stock* of connections is
attributable but none of the *flow* is.
### Why it matters
On a production cluster with 7 resource groups sharing ~14 TiDB instances, one group accounted for
**98% of all `SET`/`USE` traffic** while being only 16–20% of real queries — i.e. essentially all
connection churn came from a single application. Confirming that required using
`tidb_executor_statement_total{resource_group=...,type="Set"}` as a *proxy* for per-group connection
setup, because `tidb_server_disconnection_total` cannot be split by group.
The same gap blocked two other steps:
- Per-group KILL attribution: `tidb_server_event_total{type="kill"}` ran at ~2/s on every instance
with one instance at ~10/s, and there is no way to tell which application's connections were
being killed.
- Estimating each instance's *active* (as opposed to accumulated/idle) connection count had to be
inferred from the ratio of kills to disconnections, because neither metric is group-aware and
`conn_idle_duration_seconds` is only split by `in_txn`.
### Proposed change
Add `LblResourceGroup` to the connection-lifecycle metrics above, at minimum
`disconnection_total`, `handshake_error_total`, `event_total` and `conn_idle_duration_seconds`.
Notes on implementation
- The resource group is available on the session (`vars.ResourceGroupName`) at the points where
these counters are incremented, which is the same source `query_total` already uses at
`pkg/server/conn.go:1230`.
- `handshake_error_total` is the one case where the group may genuinely not be known yet, since the
failure can precede session initialisation. Emitting `resource_group=""` or `"default"` there is
probably acceptable, or it can be left out of scope.
- `query_total` uses a pre-built counter array for the default resource group
(`pkg/server/metrics/metrics.go:81-93`) to avoid `WithLabelValues` on the hot path. The same
pattern is available for these counters if label lookup cost is a concern. Note that array
currently has no entry for `mysql.ComChangeUser`, so COM_CHANGE_USER always takes the labelled
slow path and is reported as `type="17"` via `CmdToString`'s `strconv.Itoa` fallback — a separate
readability nit, not part of this request.
### Version
Verified on `master` (`pkg/metrics/`, fetched 2026-09-10) and `v8.5.3`; the set of metrics carrying
`LblResourceGroup` is byte-identical between the two.
Contributor guide
Research direction
Start in pkg/metrics/server.go and pkg/metrics/executor.go at the listed metric definitions, then follow the increment sites and compare pkg/server/conn.go:1230 with the resource-group handling in pkg/server/metrics/metrics.go:81-93. Determine which lifecycle paths have vars.ResourceGroupName available and verify that the requested metrics expose resource_group while retaining their existing labels; handshake-error behavior is explicitly left as a scope decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100