cockroachdb / cockroachdb/cockroach

allocator: race between decommissioning replicas and cluster node count

Open
#155,734 9 comments 0 reactions 0 assignees View on GitHub
A-kv A-kv-decom-rolling-restart A-kv-distribution branch-master C-bug P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

When no new nodes are added prior to decommission, there may be a race condition between checking for decommissioning voters in the store pool and calling `ClusterNodeCount` to get the number of live nodes.

Specifically, we might first observe no decommissioning voters at https://github.com/cockroachdb/cockroach/blob/78554d13424c610d781b47dcbc906977984b8b29/pkg/kv/kvserver/allocator.go#L687-L687,
but later see the decommissioning nodes during the subsequent `ClusterNodeCount` call at https://github.com/cockroachdb/cockroach/blob/78554d13424c610d781b47dcbc906977984b8b29/pkg/kv/kvserver/allocator.go#L690-L690.

As a result, the allocator would skip the decommission replace/remove voter check at https://github.com/cockroachdb/cockroach/blob/78554d13424c610d781b47dcbc906977984b8b29/pkg/kv/kvserver/allocator.go#L762-L762 and incorrectly proceed to the over-replicated removal check at https://github.com/cockroachdb/cockroach/blob/78554d13424c610d781b47dcbc906977984b8b29/pkg/kv/kvserver/allocator.go#L772-L772.

In this scenario, the allocator may remove a replica from the live node instead of a decommissioning node, potentially leading to loss of quorum.

More details in https://cockroachlabs.slack.com/archives/C048HDZJSAY/p1748539599115379.

Jira issue: CRDB-55599

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.