cockroachdb / cockroachdb/cockroach

kv: evaluate store liveness vs node liveness for health siganals

Open
#164,171 2 comments 0 reactions 0 assignees View on GitHub
A-kv A-kv-transactions A-leader-leases branch-master C-bug P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

With the transition to leader leases, we should evaluate places where store liveness may provide a better health signal than node liveness.

Node liveness relies on a single global range and a heartbeat mechanism that can become a bottleneck (as seen in incident), while store liveness is Raft-driven and reflects actual pairwise connectivity. Two places raised:
1. K8s readiness checks (checkReadinessForHealthCheck). The proposal is to use store liveness instead, since it's driven by Raft and only needs to confirm "this node can write."
2. Unavailable range reporting: Currently uses node liveness to determine which ranges are unavailable. Store liveness could be more accurate since it reflects actual Raft-level connectivity rather than a global heartbeat.

Some questions raised in on-call meetings: are:
(1) does store liveness more accurately capture a node's ability to serve reads and writes under leader leases
(2) how do we handle the asymmetry where different nodes have different views of store liveness
(3) can store liveness subsume the signals we currently depend on node liveness for (e.g., decommissioning state, failure detection in the allocator, unavailable range reporting)
(4) is the signal interpretable enough to be actionable in customer escalations? We should characterize both signals side-by-side and run experiments.

Jira issue: CRDB-60671

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.