resource control: clarify available_ru semantics for token preallocation
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
### Problem
`resource_manager_resource_unit_available_ru` is easy to interpret as the real-time remaining RU capacity of a resource group, but the current code exposes a narrower internal state: a snapshot of the PD-side resource group token bucket.
Relevant behavior:
- `gaugeMetrics.setGroup()` sets `available_ru` to `max(group.getRUToken(), 0)`.
- `getRUToken()` returns `RUSettings.RU.Tokens` from the server-side group token bucket.
- TiDB clients request tokens for a future period, using `avgRUPerSec * targetPeriod * needTokensAmplification - localAvailable`.
- The default target period is 5 seconds, so a modest per-second RU rate can still produce a large batched token request.
- `GroupTokenBucket.request()` and `assignSlotTokens()` may grant, trickle, or loan future tokens, then update the server-side bucket in one request.
Because of this, `available_ru` can briefly drop to 0 even when the current per-second RRU/WRU rate is below the configured fill rate. The zero can simply mean that the server bucket has just granted or loaned future-period tokens and Prometheus scraped the clamped bucket snapshot before the next refill. The current metric/dashboard does not make this distinction clear.
### Expected improvement
Improve Resource Control observability so operators can distinguish these cases:
- actual throttling or request wait caused by token shortage
- server-side token preallocation, loan, or trickle state
- service-limit throttling vs resource group fill-rate behavior
- instantaneous bucket snapshot vs sustained RU demand
Possible minimal improvements:
1. Clarify the `available_ru` metric help text and dashboard label to describe it as a server-side token-bucket balance snapshot, not real-time remaining RU capacity.
2. Expose allocation-side metrics from `RequestRU()` / `assignSlotTokens()`, such as requested tokens, granted tokens, trickle duration, and possibly loaned token amount or raw bucket balance.
3. Correlate the server-side bucket snapshot with existing client-side wait metrics, so dashboards can show whether a zero bucket balance actually caused queuing.
### Code references
- `pkg/mcs/resourcemanager/server/metrics.go`
- `gaugeMetrics.setGroup()` sets `available_ru` from `group.getRUToken()`.
- `pkg/mcs/resourcemanager/server/resource_group.go`
- `getRUToken()` returns `RUSettings.RU.Tokens`.
- `RequestRU()` calls the group token bucket and then applies the service limiter.
- `pkg/mcs/resourcemanager/server/token_buckets.go`
- `updateTokens()` performs lazy refill and balances slot tokens.
- `request()` updates the server-side bucket after assigning slot tokens.
- `assignSlotTokens()` implements grant, trickle, and loan behavior.
- `client/resource_group/controller/group_controller.go`
- `calcRequest()` computes batched future token demand.
- `client/resource_group/controller/config.go`
- `defaultTargetPeriod` is 5 seconds.
### Related
- Sub-issue of #10488, which tracks broader Resource Control observability improvements.
- Related but distinct from #9668, which addressed service-limit short-term RU spikes.
- Independent from SQL layer RU accounting semantics tracked by #10940.
Contributor guide
Research direction
Start with gaugeMetrics.setGroup() in pkg/mcs/resourcemanager/server/metrics.go, then trace getRUToken(), RequestRU(), and token_buckets.go's updateTokens(), request(), and assignSlotTokens(). Review calcRequest() and defaultTargetPeriod in the client controller to understand preallocation, then compare the available_ru snapshot with existing wait metrics. Done should make the metric semantics and token allocation or throttling state distinguishable, with tests and dashboard/help-text changes covering the chosen scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100