cockroachdb / cockroachdb/cockroach

server: account for behind replicas when computing critical nodes

Open
#119,952 1 comment 0 reactions 0 assignees View on GitHub
A-kv O-testcluster T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Its problematic when removing a node causes unavailability. The critical nodes endpoint can be used to determine which nodes aren't able to be removed without causing range unavailability.

The endpoint doesn't consider however, when replicas of a range are behind -- which can caused the loss of just 1 node to result in unavailability, e.g.

```
# killing s2 or s1 would result in unavailability
# until s3 is caught up
s1 s2 s3 (behind)
```

> Providing programatic checks would be great here, either as an endpoint or a pre-check for drain/stopping. Logging loudly in a drain would be a start, something like querying the leaseholder for each range the node has a replica for and checking whether any other replicas are in StateProbe or log is behind by more than X.

_Originally posted by @kvoli in https://github.com/cockroachdb/cockroach/issues/119523#issuecomment-1973523751_

Jira issue: CRDB-36410

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.