cockroachdb / cockroachdb/cockroach
admission: elastic CPU controller misses demand due to point-sample of hasWaitingRequests
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The elastic CPU utilization controller (`schedulerLatencyListener`) only
raises its limit when it observes both (a) p99 sched latency below target
*and* (b) `hasWaitingRequests()` returning true at the moment of its
~1Hz tick. Because the granter's `tryGrant` loop drains the queue to
empty as soon as tokens refill (`elastic_cpu_granter.go:182`), the queue
spends most of its time at zero with brief excursions when tokens are
exhausted. The 1Hz snapshot frequently lands in those zero windows even
under sustained heavy throttling.
When this happens, the controller takes the `!hasWaitingRequests` branch
in `scheduler_latency_listener.go:128-137` and *decreases* the limit
toward `inactive_point` (~12% with defaults), even though sched latency
is far under target and the host has plenty of CPU headroom.
### Evidence
Observed on a 4-node cluster running an IMPORT (src-tenant SQL
data-gen + system-tenant AddSSTable, both paced through the same
process-wide elastic CPU coordinator):
| metric | n1 | n2 | n3 | n4 |
|---|---|---|---|---|
| `admission.elastic_cpu.utilization_limit` | 0.138 | 0.135 | 0.135 | 0.139 |
| `admission.elastic_cpu.utilization` | 0.102 | 0.093 | 0.117 | 0.103 |
| `admission.scheduler_latency_listener.p99_nanos` | 92µs | 116µs | 107µs | 154µs |
| `admission.elastic_cpu.nanos_exhausted_duration` | 192s | 185s | 192s | 194s |
| `admission.elastic_cpu.over_limit_durations` (count) | 10330 | 10441 | 10709 | 11410 |
| `admission.wait_durations.elastic_cpu` (mean) | ~82ms | ~77ms | ~75ms | ~74ms |
| `admission.wait_queue_length.elastic_cpu` (sampled) | 0 | 0 | 0 | 0 |
| `sys.cpu.combined.percent-normalized` | 0.246 | 0.197 | 0.200 | 0.270 |
Limit pinned at the inactive floor (~13.5% ≈ 5% + 0.10 × (75% − 5%)),
sched p99 ~6-10× under the 1ms target, host CPU only ~20-27% — yet
~190s of cumulative token exhaustion and ~75ms mean queue wait per
request show the workload is being heavily throttled. The controller
just doesn't see it.
A fresh goroutine snapshot taken minutes after the metrics dump showed
7 stacks parked in `ElasticCPUWorkQueue.Admit`, confirming the queue
*does* fill — it just drains faster than the controller samples.
### Proposed fix
Replace the point-sample of `hasWaitingRequests()` with a sticky
\"had recent waiters\" atomic bit:
- Set the bit in `WorkQueue.Admit` at the same site that currently calls
`recordStartWait` ([work_queue.go:1001](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/work_queue.go#L1001)).
- The scheduler-latency listener atomically read-and-clears the bit each
tick instead of calling `hasWaitingRequests()`
([scheduler_latency_listener.go:103](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/scheduler_latency_listener.go#L103)).
This guarantees that any enqueue between two controller ticks is
durably visible, regardless of whether the queue subsequently drains.
### References
- [scheduler_latency_listener.go:103](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/scheduler_latency_listener.go#L103) — point-sample read site
- [work_queue.go:1214-1221](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/work_queue.go#L1214-L1221) — current `hasWaitingRequests` impl
- [work_queue.go:1001](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/work_queue.go#L1001) — natural set-site for the sticky bit
- [elastic_cpu_granter.go:182](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/elastic_cpu_granter.go#L182) — granter loop that drains the queue
Epic: none
Jira issue: CRDB-63979
Contributor guide
Assessment
This issue has not been assessed yet.