cockroachdb / cockroachdb/cockroach

kv/kvserver: underreplicated ranges from a missing store are not taken into account in a multi-store setup

Open
#123,561 1 comment 0 reactions 0 assignees View on GitHub
A-kv C-bug P-2 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

For the purpose of metrics and observability, if a node is up, all of its stores are up, from the perspective of liveness. i.e. we use node liveness, rather than store liveness.

However, in a multi-store setup, if a single store on a node is removed, or stalled for a significant period, our metrics will not show that the ranges on this store are under-replicated.

This also affects what is shown in `crdb_internal.kv_store_status`. As the store is missing, its under-replicated ranges are not visible.

For correctness, we should take store liveness into account.

Relevant code is `pkg/kv/kvserver/repica_metrics.go`, `calcRangeCounter` ([source](https://github.com/cockroachdb/cockroach/blob/21a3318243e453d9fb501119ce3872cd723c31c4/pkg/kv/kvserver/replica_metrics.go#L262)).

To reproduce:
- Configure a multi-store cluster (I used 3 nodes, each with 4 stores).
- Load enough data to place ranges on each store.
- Stop a node and restart it with one of its stores missing from the start flags (i.e. drop one of the `--store` flags).
- Prior to the `server.time_until_store_dead` interval:
- The DB console does not show under-replicated ranges
- Querying `crdb_internal.kv_store_status` will not show under-replicated ranges.
- However, querying `crdb_inernal.ranges` will show replicas still on the missing store.
- After the `server.time_until_store_dead`, as expected replicas are moved to the remaining live stores.

Jira issue: CRDB-38399

Epic CRDB-48087

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.