cockroachdb / cockroachdb/cockroach

kvserver: warn on slow raft schedule

Open
#97,910 1 comment 0 reactions 1 assignee Claimed by @pav-kv View on GitHub
C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

We saw in a recent escalation[^1] unexplained heartbeat latencies. It took a
good amount of time to figure out that the `raft.scheduler.latency` p99s were
at ~1s on the liveness leaseholder.

Similar to this warning

https://github.com/cockroachdb/cockroach/blob/736a67e0d36cc545bf74d65db069ee895ff9bea0/pkg/kv/kvserver/store_raft.go#L653-L655

we could have a (perhaps rate-limited) warning whenever it takes "long" to
schedule a raft group, as this will translate to elevated latencies in both
internal and client-facing operations.

[^1]: https://github.com/cockroachlabs/support/issues/2107

Jira issue: CRDB-24959

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.