tikv / tikv/pd

resource control: add client-side pre-throttling demand RU/s metric

Open
#10,581 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

### Problem

Currently, there is no metric that accurately reflects the **real-time RU/s demand** from clients **before** Resource Control throttling takes effect:

- **Client-side `avgRUPerSec`** (`group_controller.go`) is computed from `getRUValueFromConsumption()` — actual post-throttling consumption. When a resource group is throttled, requests wait in `Reserve()`, consumption slows, and `avgRUPerSec` only reflects the throttled rate.
- **Server-side `read_request_unit_max_per_sec` / `write_request_unit_max_per_sec`** are derived from `Consumption.RRU/WRU` reported by clients — also post-throttling values.
- **Server-side `sampled_request_unit_per_sec`** is based on `requiredToken` in `AcquireTokenBuckets`, which is `avgRUPerSec * targetPeriod * amplification - availableTokens` — not a clean demand rate, and lacks per-instance granularity.

This makes it impossible for operators to determine the true workload demand when Resource Control is actively throttling.

### Proposal

Add a new **client-side** Prometheus Gauge that tracks the EMA of demanded RU/s, sampled at the `acquireTokens()` entry point (before `Reserve()` throttling):

- **Metric**: `resource_manager_client_resource_group_demand_ru_per_sec{resource_group="..."}`
- **Data source**: the RU cost (`v`) passed to `acquireTokens()` in `group_controller.go`, which represents the true per-request demand before any token bucket throttling
- **Smoothing**: time-aware EMA (reuse the existing `movingAvgFactor` logic)

### Expected Usage

```promql
# Per-instance demand
resource_manager_client_resource_group_demand_ru_per_sec{instance="tidb-0", resource_group="default"}

# Cluster-wide demand for a resource group
sum(resource_manager_client_resource_group_demand_ru_per_sec) by (resource_group)

# Peak demand over time
max_over_time(sum(resource_manager_client_resource_group_demand_ru_per_sec) by (resource_group)[1h])
```

### Benefits

- **Accurate**: samples RU cost before throttling, reflects true workload demand
- **Per-instance**: client-side metric naturally carries `instance` label
- **Aggregatable**: `sum by` in Grafana for cluster-wide view
- **Rolling-upgrade friendly**: pure client-side change, no proto or PD server changes required

### Related

- Part of the observability improvements tracked in #10488

Contributor guide

Open the contributing guide

Research direction

Start in group_controller.go at the acquireTokens() entry point and review the existing movingAvgFactor logic for time-aware smoothing. Trace how the RU cost v is handled before Reserve() throttling. Done means a client-side Prometheus gauge named resource_manager_client_resource_group_demand_ru_per_sec reports the smoothed pre-throttling demand with the resource_group label.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.