cockroachdb / cockroachdb/cockroach
admission: GrantCoordinator.mu mutex contention
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Problem**
The CPU `GrantCoordinator` holds `GrantCoordinator.mu` while the slot and token granters call into the requester (`WorkQueue.granted` and `WorkQueue.hasWaitingRequests`), which acquires `WorkQueue.mu` and does non-trivial work.
Because of this nesting, every fast-path caller of `tryGet` and `returnGrant` can queue on `GrantCoordinator.mu` behind a goroutine that is itself waiting on `WorkQueue.mu`. Under a high rate of small KV requests, `GrantCoordinator.mu`
becomes the node's bottleneck: mutex acquisition waits reach tens of milliseconds, admitted work that has already completed cannot release its slot (releasing requires the mutex), and new work queues behind slots that are occupied only in name.
**Observed impact**
On a production cluster, a burst of small reads caused a ~60s admission stall on one node: high foreground latency and queueing (~50K work items waiting, all KV slots occupied) despite CPU peaking at only ~46% -goroutines blocked on the mutex are parked, so the node looks idle.
Related: #85471
See more in https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1788392316273769.
Epic: none
Jira issue: CRDB-67866
Contributor guide
Assessment
This issue has not been assessed yet.