cockroachdb / cockroachdb/cockroach
allocator: allocator can remove voter from the range making it unavailable
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
When replication factor for range is reduced and some replicas are located on dead nodes, allocator could remove voter and leave range without qourum.
**To Reproduce**
This situation was discovered while testing loss of quorum recovery. There is a specific step of decommissioning dead node without draining that may trigger some race.
Run 5 node cluster.
Kill nodes 4,5
Create recovery plan (needs master in run up to 23.1) `cockroach debug recover make-plan`
Apply (stage) recovery plan `cockroach debug recover apply-plan`
Perform rolling restart on nodes as instructed by staging.
Run verify `cockroach debug recovery verify'
In some cases even after all nodes were restarted there would be a range without a quorum. Moreover unavailable range is not present in original recovery plan which means it didn't lose quorum when nodes 4 and 5 were removed.
In rangelog of that range you'll see that after recovery allocator removed one of the voters from live nodes because range is overreplicated. This is happening because nodes 4 and 5 were marked as decommissioned by recovery so RF of system ranges is reduced to 3 to reflect that. But instead of removing replicas from dead nodes, live one is removed.
Jira issue: CRDB-24522
Contributor guide
Assessment
This issue has not been assessed yet.