client/resource_group: add per-instance 1s RU peak metric for resource groups
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Problem
Operators need to know the real peak RU/s a resource group actually consumes, per TiDB instance, so they can size `RU_PER_SEC` without guessing. Nothing exported today answers that:
- Server-side `resource_manager_resource_unit_{read,write}_request_unit_max_per_sec` (`pkg/mcs/resourcemanager/server/metrics.go`, `maxPerSecCostTracker`) buckets client reports by **arrival time** at PD. Clients batch consumption for 5–20s (`shouldReportConsumption`, `client/resource_group/controller/group_controller.go`), so a whole batch lands in one 1s bucket. The value is neither an upper nor a lower bound of the true peak, has no `instance` dimension, and `consumptionItem` (`manager.go`, `dispatchConsumption`) drops the client id and timestamp needed to fix it server-side.
- Server-side `resource_manager_resource_unit_sampled_request_unit_per_sec` is a 5s time-constant EMA of **requested tokens**, aggregated across clients — not consumption and not per instance.
- Client-side `resource_manager_client_request_ru_total` is accounted at request completion, so it is the right data source, but as a counter its resolution is bounded by the scrape interval (typically 15s). `irate()` over it (pingcap/tidb#70973) yields a 15s average; a 1s spike is diluted ~15x. No PromQL can recover the sub-scrape-interval peak.
### Proposal
Add a client-side gauge computed on the existing 1s state-update tick (`defaultGroupStateUpdateInterval`), next to `calcAvg` in `updateAvgRequestResourcePerSec`, which already has the per-tick consumption snapshot (`gc.run.consumption`) and the time-normalised `deltaDuration`:
- Metric: `resource_manager_client_request_ru_max_per_sec{resource_group, type="rru"|"wru"|"ru"}`; `instance` comes from Prometheus, keyspace via the existing const labels.
- Each tick: `rate = max(0, cur - last) / deltaDuration.Seconds()` per type; push into a 60-entry ring; set the gauge to the ring max.
- Sliding 60s max refreshed every second, rather than the server tracker's "publish every 20 ticks then reset", so any scrape interval ≤ 60s observes every second's peak and `max_over_time` composes exactly over longer ranges.
- Normalise by the actual tick duration (ticks can be stretched by other work on the loop) and clamp negative deltas (refunds) to 0.
- Wire into `initMetrics` / group cleanup like the other per-group metrics; add a table-driven unit test for the tracker.
Roughly 50 lines in `client/resource_group/controller/{metrics/metrics.go,group_controller.go}`; no proto or PD server change, rolling-upgrade friendly.
### Notes
- A request's RU is accounted at its completion tick, so long requests attribute their RU to the second they finish. This is inherent to the accounting model.
- `rru` and `wru` peaks can occur in different seconds; the separate `ru` series is the true total peak, not the sum of the other two.
- Throttling is enforced by the client-side limiter on roughly a 5s target-period scale with burst forced to 0 (capacity uncapped), so a 1s peak is a conservative upper bound for sizing `RU_PER_SEC`. Documenting this alongside the panel avoids over-provisioning.
### Related
- #10581 proposes a pre-throttling **demand** RU/s EMA; this issue is about the **actual consumption** peak. They are complementary.
- pingcap/tidb#70973 adds a Client RU panel over `resource_manager_client_request_ru_total`; the new gauge would back a peak panel next to it.
Contributor guide
Research direction
Start in client/resource_group/controller/metrics/metrics.go and group_controller.go, especially updateAvgRequestResourcePerSec, defaultGroupStateUpdateInterval, initMetrics, and group cleanup. Trace the existing per-tick consumption snapshot and related metrics, then add the table-driven tracker test described in the issue and run the controller package tests. Done means the per-instance client gauge exposes separate rru, wru, and ru 60-second peaks and cleans up with its group.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100