Resource Group: 1s latency spike when FillRate far exceeds actual RU consumption
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
When a resource group's `FillRate` greatly exceeds actual RU consumption (e.g., FillRate=1,000,000 vs ~200 RU/s), requests intermittently experience **~1s blocking** due to the client-side token limiter entering a starved state.
### Reproduction
1. Create a resource group with a high `FillRate` (e.g., 1,000,000 RU/s)
2. Run a low-traffic workload (~200 RU/s)
3. Observe periodic ~1s latency spikes on SQL requests
### Root Cause
In non-trickle mode (RU abundant), the server intentionally returns `FillRate=0` — the client consumes from a granted token pool without maintaining a local rate limiter. When tokens deplete and the limiter transitions from non-trickle (`rate=0`) to needing more tokens:
1. `reserveN()` with `limit=0` → `durationFromTokens()` returns `InfDuration` → reservation fails immediately
2. `notify()` triggers an async RPC to the server (~50-100ms round trip)
3. `acquireTokens()` enters a retry loop: **20 retries × 50ms blind `time.Sleep`** = up to **1000ms**
4. The async token refresh completes in ~50-100ms via `Reconfigure()`, but the retry loop has no way to observe this — it keeps sleeping
```go
for range gc.mainCfg.WaitRetryTimes { // 20
res = counter.limiter.Reserve(...)
if d, err = WaitReservations(...); err == nil {
break
}
time.Sleep(gc.mainCfg.WaitRetryInterval) // 50ms, blind
}
```
### Impact
- **P95/P99 latency spikes** of ~1s under low-traffic workloads with high FillRate configurations
- More pronounced with single-client setups where slot deletion happens every period
### Expected Behavior
When tokens are refreshed via `Reconfigure()` (~50-100ms), the retry loop should wake up immediately and proceed, not continue blind-sleeping for up to 1s.
Contributor guide
Assessment
This issue has not been assessed yet.