cockroachdb / cockroachdb/cockroach

admission: GrantCoordinator.mu mutex contention

Open
#174,668 0 comments 0 reactions 1 assignee Claimed by @wenyihu6 View on GitHub
A-admission-control C-performance O-agent O-support T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Problem**

The CPU `GrantCoordinator` holds `GrantCoordinator.mu` while the slot and token granters call into the requester (`WorkQueue.granted` and `WorkQueue.hasWaitingRequests`), which acquires `WorkQueue.mu` and does non-trivial work.

Because of this nesting, every fast-path caller of `tryGet` and `returnGrant` can queue on `GrantCoordinator.mu` behind a goroutine that is itself waiting on `WorkQueue.mu`. Under a high rate of small KV requests, `GrantCoordinator.mu`
becomes the node's bottleneck: mutex acquisition waits reach tens of milliseconds, admitted work that has already completed cannot release its slot (releasing requires the mutex), and new work queues behind slots that are occupied only in name.

**Observed impact**

On a production cluster, a burst of small reads caused a ~60s admission stall on one node: high foreground latency and queueing (~50K work items waiting, all KV slots occupied) despite CPU peaking at only ~46% -goroutines blocked on the mutex are parked, so the node looks idle.

Related: #85471
See more in https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1788392316273769.

Epic: none
Jira issue: CRDB-67866

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.