tikv / tikv/pd

Resource Group: 1s latency spike when FillRate far exceeds actual RU consumption

Open
#10,251 0 comments 0 reactions 0 assignees View on GitHub
affects-7.1 affects-7.5 affects-8.1 affects-8.5 severity/major type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

When a resource group's `FillRate` greatly exceeds actual RU consumption (e.g., FillRate=1,000,000 vs ~200 RU/s), requests intermittently experience **~1s blocking** due to the client-side token limiter entering a starved state.

### Reproduction

1. Create a resource group with a high `FillRate` (e.g., 1,000,000 RU/s)
2. Run a low-traffic workload (~200 RU/s)
3. Observe periodic ~1s latency spikes on SQL requests

### Root Cause

In non-trickle mode (RU abundant), the server intentionally returns `FillRate=0` — the client consumes from a granted token pool without maintaining a local rate limiter. When tokens deplete and the limiter transitions from non-trickle (`rate=0`) to needing more tokens:

1. `reserveN()` with `limit=0` → `durationFromTokens()` returns `InfDuration` → reservation fails immediately
2. `notify()` triggers an async RPC to the server (~50-100ms round trip)
3. `acquireTokens()` enters a retry loop: **20 retries × 50ms blind `time.Sleep`** = up to **1000ms**
4. The async token refresh completes in ~50-100ms via `Reconfigure()`, but the retry loop has no way to observe this — it keeps sleeping

```go
for range gc.mainCfg.WaitRetryTimes { // 20
res = counter.limiter.Reserve(...)
if d, err = WaitReservations(...); err == nil {
break
}
time.Sleep(gc.mainCfg.WaitRetryInterval) // 50ms, blind
}
```

### Impact

- **P95/P99 latency spikes** of ~1s under low-traffic workloads with high FillRate configurations
- More pronounced with single-client setups where slot deletion happens every period

### Expected Behavior

When tokens are refreshed via `Reconfigure()` (~50-100ms), the retry loop should wake up immediately and proceed, not continue blind-sleeping for up to 1s.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.