cockroachdb / cockroachdb/cockroach
kvserver: warn on slow raft schedule
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
We saw in a recent escalation[^1] unexplained heartbeat latencies. It took a
good amount of time to figure out that the `raft.scheduler.latency` p99s were
at ~1s on the liveness leaseholder.
Similar to this warning
https://github.com/cockroachdb/cockroach/blob/736a67e0d36cc545bf74d65db069ee895ff9bea0/pkg/kv/kvserver/store_raft.go#L653-L655
we could have a (perhaps rate-limited) warning whenever it takes "long" to
schedule a raft group, as this will translate to elevated latencies in both
internal and client-facing operations.
[^1]: https://github.com/cockroachlabs/support/issues/2107
Jira issue: CRDB-24959
Contributor guide
Assessment
This issue has not been assessed yet.