cockroachdb / cockroachdb/cockroach

allocator: allocator can remove voter from the range making it unavailable

Open
#97,114 0 comments 0 reactions 0 assignees View on GitHub
A-kv A-kv-distribution C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

When replication factor for range is reduced and some replicas are located on dead nodes, allocator could remove voter and leave range without qourum.

**To Reproduce**

This situation was discovered while testing loss of quorum recovery. There is a specific step of decommissioning dead node without draining that may trigger some race.

Run 5 node cluster.
Kill nodes 4,5
Create recovery plan (needs master in run up to 23.1) `cockroach debug recover make-plan`
Apply (stage) recovery plan `cockroach debug recover apply-plan`
Perform rolling restart on nodes as instructed by staging.
Run verify `cockroach debug recovery verify'

In some cases even after all nodes were restarted there would be a range without a quorum. Moreover unavailable range is not present in original recovery plan which means it didn't lose quorum when nodes 4 and 5 were removed.

In rangelog of that range you'll see that after recovery allocator removed one of the voters from live nodes because range is overreplicated. This is happening because nodes 4 and 5 were marked as decommissioned by recovery so RF of system ranges is reduced to 3 to reflect that. But instead of removing replicas from dead nodes, live one is removed.

Jira issue: CRDB-24522

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.