cockroachdb / cockroachdb/cockroach

kvserver: introduce and use TimedMutex for raftMu

Open
#99,473 3 comments 0 reactions 0 assignees View on GitHub
C-enhancement O-support P-3 quality-friday T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

We've had a few support escalations where high raft scheduler latencies were observed. These were only high in the top percentiles, meaning the explanation is that the scheduler had enough slack, but it wasn't able to schedule some replicas in a timely manner, likely because their raftMu was held by "someone". In the cases below, the raft processing itself was not that "someone". In the first case, it turned out to be a side effect of a liveness lease acquisition, and has since been addressed, but the other incident is currently not RCAed.

x-ref https://github.com/cockroachlabs/support/issues/2107.
x-ref https://github.com/cockroachlabs/support/issues/2182.

**Describe the solution you'd like**

raftMu is a long critical section and so we can afford some overhead. Implement something like the prototype in https://github.com/cockroachdb/cockroach/pull/98560 to get raftMu metrics and also print the stack of operations that release raftMu after having held it for (say) >1s. (We could exclude the raft processing goroutine from this warning since it already emits such messages with a more detailed breakdown of where it spent its time).

Jira issue: CRDB-25894

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.