cockroachdb / cockroachdb/cockroach

kvserver: LoQ when reducing quorum with suspect nodes

Open
#152,604 8 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution branch-master C-bug P-2 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When downsizing zone config voters from 5 to 3, we observed that the allocator can remove a wrong replica. Instead of removing one that is no longer mentioned in the constraints, it removes one of the "required" replicas. If this happens while the cluster is in a vulnerable state with only the 3 replicas alive, we lose quorum.

Crucially, this bug only reproduces in the first 5 minutes (`server.time_until_store_dead` cluster setting) of nodes being dead. This makes the allocator take the [RemoveVoter](https://github.com/cockroachdb/cockroach/blob/b168254ff7e2165151693eaf56d0efd0ee93ab11/pkg/kv/kvserver/allocator/plan/replicate.go#L284-L285) path (rather than `RemoveDeadVoter`).

**Version:** v25.2, likely on other branches too.

**Reproduction**

```bash
# 1. Checkout release-25.2 and build cockroach and roachprod
...

# 2. Create a cluster with 9 nodes locally.
roachprod create -n9 local
roachprod put local ./cockroach
roachprod start local --args="--advertise-addr=127.0.0.1" # the flag works around issue #149469

# 3. Make it 3 regions, 3 nodes in each. The fastest way I found is:
# stop the cluster, and manually change the --locality flag in each
# node's cockroach.sh script.
roachprod stop local
cd ~/local

sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg1,zone=a/g' 1/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg1,zone=b/g' 2/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg1,zone=c/g' 3/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg2,zone=a/g' 4/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg2,zone=b/g' 5/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg2,zone=c/g' 6/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg3,zone=a/g' 7/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg3,zone=b/g' 8/cockroach.sh
sed -i '' -e 's/--locality=.*/--locality=cloud=local,region=reg3,zone=c/g' 9/cockroach.sh

# don't "roachprod start" because it will reset the cockroach.sh scripts
for i in {1..9}; do
cd $i
./cockroach.sh
cd -
done

---

# 1. Create a table.
roachprod sql local:1
>
CREATE DATABASE d1;
USE d1;
CREATE TABLE t1 (i INT PRIMARY KEY);

# 2. Configure it for 2+2+1 replication and leases in the first 2 regions.
ALTER DATABASE d1 CONFIGURE ZONE USING
num_replicas = 5, num_voters = 5,
constraints = '{"+region=reg1": 2, "+region=reg2": 2, "+region=reg3": 1}',
voter_constraints = '{"+region=reg1": 2, "+region=reg2": 2, "+region=reg3": 1}',
lease_preferences = '[[-region=reg3]]';

# 3. Populate it with some data, and create a bunch of splits.
# The bug needs many splits to have a better reproduction chance.
INSERT INTO t1(i) SELECT generate_series(1,1000);
ALTER TABLE t1 SPLIT AT SELECT i FROM t1;

# 4. Check with the CRDB console, wait until the data is fully
# replicated ("underreplicated ranges" is zero).
roachprod adminurl local:1

---

# 1. Kill the reg2 region.
ps aux | grep reg2
kill -9

# 2. Within the next 5 minutes, change the zone config to RF=3 with 2+1 replication.
ALTER DATABASE d1 CONFIGURE ZONE USING
num_replicas = 3, num_voters = 3,
constraints = '{"+region=reg1": 2, "+region=reg3": 1}',
voter_constraints = '{"+region=reg1": 2, "+region=reg3": 1}',
lease_preferences = '[[-region=reg3]]';

# 3. Try to read from this table. Boom! "lost quorum" errors
SELECT COUNT(*) FROM t1;

ERROR: replica unavailable: (n4,s4):6 unable to serve request to r75:/Table/106{-/1/1}
[(n4,s4):6, (n2,s2):7, (n9,s9):8, (n5,s5):9VOTER_DEMOTING_LEARNER, (n1,s1):10, next=11, gen=33]:
lost quorum (down: (n9,s9):8,(n1,s1):10);
closed timestamp: 1755842070.279712294,0 (2025-08-22 05:54:30);
raft status: {"id":"6","term":8,"vote":"6","commit":192,"lead":"0","leadEpoch":"0","raftState":"StateFollower","applied":192,"progress":{},"leadtransferee":"0"}:
replica has been leaderless for 1m0s
```

More context: https://cockroachlabs.slack.com/archives/G01G8LK77DK/p1756177648619009

Jira issue: CRDB-53940

Contributor guide

Open the contributing guide

Research direction

Start in pkg/kv/kvserver/allocator/plan/replicate.go at the RemoveVoter and RemoveDeadVoter paths referenced by the issue. Reproduce the v25.2 scenario with a nine-node, three-region cluster, then inspect why downsizing during the first five minutes selects a required replica. Done means the allocator removes the replica no longer required without causing lost quorum.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.