tikv / tikv/pd

resource control: clarify available_ru semantics for token preallocation

Open
#10,941 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

### Problem

`resource_manager_resource_unit_available_ru` is easy to interpret as the real-time remaining RU capacity of a resource group, but the current code exposes a narrower internal state: a snapshot of the PD-side resource group token bucket.

Relevant behavior:

- `gaugeMetrics.setGroup()` sets `available_ru` to `max(group.getRUToken(), 0)`.
- `getRUToken()` returns `RUSettings.RU.Tokens` from the server-side group token bucket.
- TiDB clients request tokens for a future period, using `avgRUPerSec * targetPeriod * needTokensAmplification - localAvailable`.
- The default target period is 5 seconds, so a modest per-second RU rate can still produce a large batched token request.
- `GroupTokenBucket.request()` and `assignSlotTokens()` may grant, trickle, or loan future tokens, then update the server-side bucket in one request.

Because of this, `available_ru` can briefly drop to 0 even when the current per-second RRU/WRU rate is below the configured fill rate. The zero can simply mean that the server bucket has just granted or loaned future-period tokens and Prometheus scraped the clamped bucket snapshot before the next refill. The current metric/dashboard does not make this distinction clear.

### Expected improvement

Improve Resource Control observability so operators can distinguish these cases:

- actual throttling or request wait caused by token shortage
- server-side token preallocation, loan, or trickle state
- service-limit throttling vs resource group fill-rate behavior
- instantaneous bucket snapshot vs sustained RU demand

Possible minimal improvements:

1. Clarify the `available_ru` metric help text and dashboard label to describe it as a server-side token-bucket balance snapshot, not real-time remaining RU capacity.
2. Expose allocation-side metrics from `RequestRU()` / `assignSlotTokens()`, such as requested tokens, granted tokens, trickle duration, and possibly loaned token amount or raw bucket balance.
3. Correlate the server-side bucket snapshot with existing client-side wait metrics, so dashboards can show whether a zero bucket balance actually caused queuing.

### Code references

- `pkg/mcs/resourcemanager/server/metrics.go`
- `gaugeMetrics.setGroup()` sets `available_ru` from `group.getRUToken()`.
- `pkg/mcs/resourcemanager/server/resource_group.go`
- `getRUToken()` returns `RUSettings.RU.Tokens`.
- `RequestRU()` calls the group token bucket and then applies the service limiter.
- `pkg/mcs/resourcemanager/server/token_buckets.go`
- `updateTokens()` performs lazy refill and balances slot tokens.
- `request()` updates the server-side bucket after assigning slot tokens.
- `assignSlotTokens()` implements grant, trickle, and loan behavior.
- `client/resource_group/controller/group_controller.go`
- `calcRequest()` computes batched future token demand.
- `client/resource_group/controller/config.go`
- `defaultTargetPeriod` is 5 seconds.

### Related

- Sub-issue of #10488, which tracks broader Resource Control observability improvements.
- Related but distinct from #9668, which addressed service-limit short-term RU spikes.
- Independent from SQL layer RU accounting semantics tracked by #10940.

Contributor guide

Open the contributing guide

Research direction

Start with gaugeMetrics.setGroup() in pkg/mcs/resourcemanager/server/metrics.go, then trace getRUToken(), RequestRU(), and token_buckets.go's updateTokens(), request(), and assignSlotTokens(). Review calcRequest() and defaultTargetPeriod in the client controller to understand preallocation, then compare the available_ru snapshot with existing wait metrics. Done should make the metric semantics and token allocation or throttling state distinguishable, with tests and dashboard/help-text changes covering the chosen scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.