cockroachdb / cockroachdb/cockroach
kvserver: LoQ when reducing quorum with suspect nodes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When downsizing zone config voters from 5 to 3, we observed that the allocator can remove a wrong replica. Instead of removing one that is no longer mentioned in the constraints, it removes one of the "required" replicas. If this happens while the cluster is in a vulnerable state with only the 3 replicas alive, we lose quorum.
Crucially, this bug only reproduces in the first 5 minutes (`server.time_until_store_dead` cluster setting) of nodes being dead. This makes the allocator take the [RemoveVoter](https://github.com/cockroachdb/cockroach/blob/b168254ff7e2165151693eaf56d0efd0ee93ab11/pkg/kv/kvserver/allocator/plan/replicate.go#L284-L285) path (rather than `RemoveDeadVoter`).
**Version:** v25.2, likely on other branches too.
**Reproduction**
```bash
# 1. Checkout release-25.2 and build cockroach and roachprod
...
# 2. Create a cluster with 9 nodes locally.
roachprod create -n9 local
roachprod put local ./cockroach
roachprod start local --args="--advertise-addr=127.0.0.1" # the flag works around issue #149469
# 3. Make it 3 regions, 3 nodes in each. The fastest way I found is:
# stop the cluster, and manually change the --locality flag in each
# node's cockroach.sh script.
roachprod stop local
cd ~/local
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg1,zone=a/g' 1/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg1,zone=b/g' 2/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg1,zone=c/g' 3/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg2,zone=a/g' 4/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg2,zone=b/g' 5/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg2,zone=c/g' 6/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg3,zone=a/g' 7/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg3,zone=b/g' 8/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg3,zone=c/g' 9/cockroach.sh
# don't "roachprod start" because it will reset the cockroach.sh scripts
for i in {1..9}; do
cd $i
./cockroach.sh
cd -
done
---
# 1. Create a table.
roachprod sql local:1
>
CREATE DATABASE d1;
USE d1;
CREATE TABLE t1 (i INT PRIMARY KEY);
# 2. Configure it for 2+2+1 replication and leases in the first 2 regions.
ALTER DATABASE d1 CONFIGURE ZONE USING
num_replicas = 5, num_voters = 5,
constraints = '{"+region=reg1": 2, "+region=reg2": 2, "+region=reg3": 1}',
voter_constraints = '{"+region=reg1": 2, "+region=reg2": 2, "+region=reg3": 1}',
lease_preferences = '[[-region=reg3]]';
# 3. Populate it with some data, and create a bunch of splits.
# The bug needs many splits to have a better reproduction chance.
INSERT INTO t1(i) SELECT generate_series(1,1000);
ALTER TABLE t1 SPLIT AT SELECT i FROM t1;
# 4. Check with the CRDB console, wait until the data is fully
# replicated ("underreplicated ranges" is zero).
roachprod adminurl local:1
---
# 1. Kill the reg2 region.
ps aux | grep reg2
kill -9
# 2. Within the next 5 minutes, change the zone config to RF=3 with 2+1 replication.
ALTER DATABASE d1 CONFIGURE ZONE USING
num_replicas = 3, num_voters = 3,
constraints = '{"+region=reg1": 2, "+region=reg3": 1}',
voter_constraints = '{"+region=reg1": 2, "+region=reg3": 1}',
lease_preferences = '[[-region=reg3]]';
# 3. Try to read from this table. Boom! "lost quorum" errors
SELECT COUNT(*) FROM t1;
ERROR: replica unavailable: (n4,s4):6 unable to serve request to r75:/Table/106{-/1/1}
[(n4,s4):6, (n2,s2):7, (n9,s9):8, (n5,s5):9VOTER_DEMOTING_LEARNER, (n1,s1):10, next=11, gen=33]:
lost quorum (down: (n9,s9):8,(n1,s1):10);
closed timestamp: 1755842070.279712294,0 (2025-08-22 05:54:30);
raft status: {"id":"6","term":8,"vote":"6","commit":192,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":192,"progress":{},"leadtransferee":"0"}:
replica has been leaderless for 1m0s
```
More context: https://cockroachlabs.slack.com/archives/G01G8LK77DK/p1756177648619009
Jira issue: CRDB-53940
Contributor guide
Research direction
Start in pkg/kv/kvserver/allocator/plan/replicate.go at the RemoveVoter and RemoveDeadVoter paths referenced by the issue. Reproduce the v25.2 scenario with a nine-node, three-region cluster, then inspect why downsizing during the first five minutes selects a required replica. Done means the allocator removes the replica no longer required without causing lost quorum.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100