cockroachdb / cockroachdb/cockroach

admission: elastic CPU controller misses demand due to point-sample of hasWaitingRequests

Open
#170,400 2 comments 0 reactions 1 assignee Claimed by @wenyihu6 View on GitHub
A-admission-control C-bug O-agent P-3 T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The elastic CPU utilization controller (`schedulerLatencyListener`) only
raises its limit when it observes both (a) p99 sched latency below target
*and* (b) `hasWaitingRequests()` returning true at the moment of its
~1Hz tick. Because the granter's `tryGrant` loop drains the queue to
empty as soon as tokens refill (`elastic_cpu_granter.go:182`), the queue
spends most of its time at zero with brief excursions when tokens are
exhausted. The 1Hz snapshot frequently lands in those zero windows even
under sustained heavy throttling.

When this happens, the controller takes the `!hasWaitingRequests` branch
in `scheduler_latency_listener.go:128-137` and *decreases* the limit
toward `inactive_point` (~12% with defaults), even though sched latency
is far under target and the host has plenty of CPU headroom.

### Evidence

Observed on a 4-node cluster running an IMPORT (src-tenant SQL
data-gen + system-tenant AddSSTable, both paced through the same
process-wide elastic CPU coordinator):

| metric | n1 | n2 | n3 | n4 |
|---|---|---|---|---|
| `admission.elastic_cpu.utilization_limit` | 0.138 | 0.135 | 0.135 | 0.139 |
| `admission.elastic_cpu.utilization` | 0.102 | 0.093 | 0.117 | 0.103 |
| `admission.scheduler_latency_listener.p99_nanos` | 92µs | 116µs | 107µs | 154µs |
| `admission.elastic_cpu.nanos_exhausted_duration` | 192s | 185s | 192s | 194s |
| `admission.elastic_cpu.over_limit_durations` (count) | 10330 | 10441 | 10709 | 11410 |
| `admission.wait_durations.elastic_cpu` (mean) | ~82ms | ~77ms | ~75ms | ~74ms |
| `admission.wait_queue_length.elastic_cpu` (sampled) | 0 | 0 | 0 | 0 |
| `sys.cpu.combined.percent-normalized` | 0.246 | 0.197 | 0.200 | 0.270 |

Limit pinned at the inactive floor (~13.5% ≈ 5% + 0.10 × (75% − 5%)),
sched p99 ~6-10× under the 1ms target, host CPU only ~20-27% — yet
~190s of cumulative token exhaustion and ~75ms mean queue wait per
request show the workload is being heavily throttled. The controller
just doesn't see it.

A fresh goroutine snapshot taken minutes after the metrics dump showed
7 stacks parked in `ElasticCPUWorkQueue.Admit`, confirming the queue
*does* fill — it just drains faster than the controller samples.

### Proposed fix

Replace the point-sample of `hasWaitingRequests()` with a sticky
\"had recent waiters\" atomic bit:

- Set the bit in `WorkQueue.Admit` at the same site that currently calls
`recordStartWait` ([work_queue.go:1001](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/work_queue.go#L1001)).
- The scheduler-latency listener atomically read-and-clears the bit each
tick instead of calling `hasWaitingRequests()`
([scheduler_latency_listener.go:103](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/scheduler_latency_listener.go#L103)).

This guarantees that any enqueue between two controller ticks is
durably visible, regardless of whether the queue subsequently drains.

### References

- [scheduler_latency_listener.go:103](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/scheduler_latency_listener.go#L103) — point-sample read site
- [work_queue.go:1214-1221](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/work_queue.go#L1214-L1221) — current `hasWaitingRequests` impl
- [work_queue.go:1001](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/work_queue.go#L1001) — natural set-site for the sticky bit
- [elastic_cpu_granter.go:182](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/elastic_cpu_granter.go#L182) — granter loop that drains the queue

Epic: none

Jira issue: CRDB-63979

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.