cockroachdb / cockroachdb/cockroach
server: account for behind replicas when computing critical nodes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Its problematic when removing a node causes unavailability. The critical nodes endpoint can be used to determine which nodes aren't able to be removed without causing range unavailability.
The endpoint doesn't consider however, when replicas of a range are behind -- which can caused the loss of just 1 node to result in unavailability, e.g.
```
# killing s2 or s1 would result in unavailability
# until s3 is caught up
s1 s2 s3 (behind)
```
> Providing programatic checks would be great here, either as an endpoint or a pre-check for drain/stopping. Logging loudly in a drain would be a start, something like querying the leaseholder for each range the node has a replica for and checking whether any other replicas are in StateProbe or log is behind by more than X.
_Originally posted by @kvoli in https://github.com/cockroachdb/cockroach/issues/119523#issuecomment-1973523751_
Jira issue: CRDB-36410
Contributor guide
Assessment
This issue has not been assessed yet.