tikv / tikv/pd

client/resource_group: add per-instance 1s RU peak metric for resource groups

Open
#11,256 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

### Problem

Operators need to know the real peak RU/s a resource group actually consumes, per TiDB instance, so they can size `RU_PER_SEC` without guessing. Nothing exported today answers that:

- Server-side `resource_manager_resource_unit_{read,write}_request_unit_max_per_sec` (`pkg/mcs/resourcemanager/server/metrics.go`, `maxPerSecCostTracker`) buckets client reports by **arrival time** at PD. Clients batch consumption for 5–20s (`shouldReportConsumption`, `client/resource_group/controller/group_controller.go`), so a whole batch lands in one 1s bucket. The value is neither an upper nor a lower bound of the true peak, has no `instance` dimension, and `consumptionItem` (`manager.go`, `dispatchConsumption`) drops the client id and timestamp needed to fix it server-side.
- Server-side `resource_manager_resource_unit_sampled_request_unit_per_sec` is a 5s time-constant EMA of **requested tokens**, aggregated across clients — not consumption and not per instance.
- Client-side `resource_manager_client_request_ru_total` is accounted at request completion, so it is the right data source, but as a counter its resolution is bounded by the scrape interval (typically 15s). `irate()` over it (pingcap/tidb#70973) yields a 15s average; a 1s spike is diluted ~15x. No PromQL can recover the sub-scrape-interval peak.

### Proposal

Add a client-side gauge computed on the existing 1s state-update tick (`defaultGroupStateUpdateInterval`), next to `calcAvg` in `updateAvgRequestResourcePerSec`, which already has the per-tick consumption snapshot (`gc.run.consumption`) and the time-normalised `deltaDuration`:

- Metric: `resource_manager_client_request_ru_max_per_sec{resource_group, type="rru"|"wru"|"ru"}`; `instance` comes from Prometheus, keyspace via the existing const labels.
- Each tick: `rate = max(0, cur - last) / deltaDuration.Seconds()` per type; push into a 60-entry ring; set the gauge to the ring max.
- Sliding 60s max refreshed every second, rather than the server tracker's "publish every 20 ticks then reset", so any scrape interval ≤ 60s observes every second's peak and `max_over_time` composes exactly over longer ranges.
- Normalise by the actual tick duration (ticks can be stretched by other work on the loop) and clamp negative deltas (refunds) to 0.
- Wire into `initMetrics` / group cleanup like the other per-group metrics; add a table-driven unit test for the tracker.

Roughly 50 lines in `client/resource_group/controller/{metrics/metrics.go,group_controller.go}`; no proto or PD server change, rolling-upgrade friendly.

### Notes

- A request's RU is accounted at its completion tick, so long requests attribute their RU to the second they finish. This is inherent to the accounting model.
- `rru` and `wru` peaks can occur in different seconds; the separate `ru` series is the true total peak, not the sum of the other two.
- Throttling is enforced by the client-side limiter on roughly a 5s target-period scale with burst forced to 0 (capacity uncapped), so a 1s peak is a conservative upper bound for sizing `RU_PER_SEC`. Documenting this alongside the panel avoids over-provisioning.

### Related

- #10581 proposes a pre-throttling **demand** RU/s EMA; this issue is about the **actual consumption** peak. They are complementary.
- pingcap/tidb#70973 adds a Client RU panel over `resource_manager_client_request_ru_total`; the new gauge would back a peak panel next to it.

Contributor guide

Open the contributing guide

Research direction

Start in client/resource_group/controller/metrics/metrics.go and group_controller.go, especially updateAvgRequestResourcePerSec, defaultGroupStateUpdateInterval, initMetrics, and group cleanup. Trace the existing per-tick consumption snapshot and related metrics, then add the table-driven tracker test described in the issue and run the controller package tests. Done means the per-instance client gauge exposes separate rru, wru, and ru 60-second peaks and cleans up with its group.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.