cockroachdb / cockroachdb/cockroach
kv/kvserver: underreplicated ranges from a missing store are not taken into account in a multi-store setup
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
For the purpose of metrics and observability, if a node is up, all of its stores are up, from the perspective of liveness. i.e. we use node liveness, rather than store liveness.
However, in a multi-store setup, if a single store on a node is removed, or stalled for a significant period, our metrics will not show that the ranges on this store are under-replicated.
This also affects what is shown in `crdb_internal.kv_store_status`. As the store is missing, its under-replicated ranges are not visible.
For correctness, we should take store liveness into account.
Relevant code is `pkg/kv/kvserver/repica_metrics.go`, `calcRangeCounter` ([source](https://github.com/cockroachdb/cockroach/blob/21a3318243e453d9fb501119ce3872cd723c31c4/pkg/kv/kvserver/replica_metrics.go#L262)).
To reproduce:
- Configure a multi-store cluster (I used 3 nodes, each with 4 stores).
- Load enough data to place ranges on each store.
- Stop a node and restart it with one of its stores missing from the start flags (i.e. drop one of the `--store` flags).
- Prior to the `server.time_until_store_dead` interval:
- The DB console does not show under-replicated ranges
- Querying `crdb_internal.kv_store_status` will not show under-replicated ranges.
- However, querying `crdb_inernal.ranges` will show replicas still on the missing store.
- After the `server.time_until_store_dead`, as expected replicas are moved to the remaining live stores.
Jira issue: CRDB-38399
Epic CRDB-48087
Contributor guide
Assessment
This issue has not been assessed yet.