cockroachdb / cockroachdb/cockroach

kvserver: improve ReplicaUnavailableError

Open
#94,691 2 comments 0 reactions 0 assignees View on GitHub
A-kv-observability C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Loss of quorum-scenarios can be difficult to detect. Currently, the signals are mostly around "slowless", e.g. replica circuit breakers reporting slow proposals, requests reporting slow latch acquisition, requests and queries timing out, etc. We have also seen situations where we lost quorum on the liveness range, and thus didn't have access to the DB console or any other debugging tools, which took a long time to resolve (https://github.com/cockroachlabs/support/issues/1963).

We should warn loudly, both in logs and the DB console/metrics, when unquiesced ranges are unable to acquire a Raft leader for some time. This is a much more specific signal for quorum loss, which immediately rules out lots of other causes of "slowness".

Jira issue: CRDB-23087

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.