cockroachdb / cockroachdb/cockroach
allocator: race between decommissioning replicas and cluster node count
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
When no new nodes are added prior to decommission, there may be a race condition between checking for decommissioning voters in the store pool and calling `ClusterNodeCount` to get the number of live nodes.
Specifically, we might first observe no decommissioning voters at https://github.com/cockroachdb/cockroach/blob/78554d13424c610d781b47dcbc906977984b8b29/pkg/kv/kvserver/allocator.go#L687-L687,
but later see the decommissioning nodes during the subsequent `ClusterNodeCount` call at https://github.com/cockroachdb/cockroach/blob/78554d13424c610d781b47dcbc906977984b8b29/pkg/kv/kvserver/allocator.go#L690-L690.
As a result, the allocator would skip the decommission replace/remove voter check at https://github.com/cockroachdb/cockroach/blob/78554d13424c610d781b47dcbc906977984b8b29/pkg/kv/kvserver/allocator.go#L762-L762 and incorrectly proceed to the over-replicated removal check at https://github.com/cockroachdb/cockroach/blob/78554d13424c610d781b47dcbc906977984b8b29/pkg/kv/kvserver/allocator.go#L772-L772.
In this scenario, the allocator may remove a replica from the live node instead of a decommissioning node, potentially leading to loss of quorum.
More details in https://cockroachlabs.slack.com/archives/C048HDZJSAY/p1748539599115379.
Jira issue: CRDB-55599
Contributor guide
Assessment
This issue has not been assessed yet.