cockroachdb / cockroachdb/cockroach
kv: evaluate store liveness vs node liveness for health siganals
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
With the transition to leader leases, we should evaluate places where store liveness may provide a better health signal than node liveness.
Node liveness relies on a single global range and a heartbeat mechanism that can become a bottleneck (as seen in incident), while store liveness is Raft-driven and reflects actual pairwise connectivity. Two places raised:
1. K8s readiness checks (checkReadinessForHealthCheck). The proposal is to use store liveness instead, since it's driven by Raft and only needs to confirm "this node can write."
2. Unavailable range reporting: Currently uses node liveness to determine which ranges are unavailable. Store liveness could be more accurate since it reflects actual Raft-level connectivity rather than a global heartbeat.
Some questions raised in on-call meetings: are:
(1) does store liveness more accurately capture a node's ability to serve reads and writes under leader leases
(2) how do we handle the asymmetry where different nodes have different views of store liveness
(3) can store liveness subsume the signals we currently depend on node liveness for (e.g., decommissioning state, failure detection in the allocator, unavailable range reporting)
(4) is the signal interpretable enough to be actionable in customer escalations? We should characterize both signals side-by-side and run experiments.
Jira issue: CRDB-60671
Contributor guide
Assessment
This issue has not been assessed yet.