cockroachdb / cockroachdb/cockroach

allocator: quantified zone configuration constraints don't prohibit unconstrained replication

Open
#131,289 2 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution branch-master branch-release-23.1 branch-release-23.2 branch-release-24.1 branch-release-24.2 branch-release-24.3 C-bug P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When setting zone config constraints, a common pattern with 5x replication is assign two regions, a primary and secondary, which are preferred as leaseholders, with a third region holding a single replica, enabling regional fail over.

If the primary and secondary region fail slowly enough, with enough time to upreplicate the dead replicas, then its possible that ranges will be misreplicated to have 3 replicas in the region which should only have one. While this may be desirable, it is problematic if users wish to have **exactly one** replica in the third region.

e.g.

```bash
roachprod create local -n 9
roachprod start local:1 --args="--locality=region=primary,zone=zone-1"
roachprod start local:2 --args="--locality=region=primary,zone=zone-3"
roachprod start local:3 --args="--locality=region=primary,zone=zone-3"
roachprod start local:4 --args="--locality=region=secondary,zone=zone-1"
roachprod start local:5 --args="--locality=region=secondary,zone=zone-3"
roachprod start local:6 --args="--locality=region=secondary,zone=zone-3"
roachprod start local:7 --args="--locality=region=tertiary,zone=zone-1"
roachprod start local:8 --args="--locality=region=tertiary,zone=zone-3"
roachprod start local:9 --args="--locality=region=tertiary,zone=zone-3"

roachprod sql local:1 -- -e "SET CLUSTER SETTING server.time_until_store_dead = '90s';"
roachprod sql local:1 -- -e "CREATE DATABASE fd;"
roachprod sql local:1 -- -e "ALTER DATABASE fd CONFIGURE ZONE USING num_replicas = 5, num_voters = 5, constraints = '{"+region=primary": 2, "+region=secondary": 2, "+region=tertiary": 1}', voter_constraints = '{"+region=primary": 2, "+region=secondary": 2, "+region=tertiary": 1}', lease_preferences = '[[+region=primary],[+region=secondary]]';"
roachprod sql local:1 -- -e "CREATE TABLE fd.t(i INT);"
roachprod sql local:1 -- -e "INSERT INTO fd.t(i) select generate_series(1,10);"
roachprod sql local:1 -- -e "ALTER TABLE fd.t SPLIT AT SELECT i FROM fd.t;"

sleep 180
roachprod stop local:1-3
sleep 180
roachprod stop local:4
sleep 180
roachprod stop local:5
# No need to sleep again, there will be 3 replicas on region=tertiary.
roachprod stop local:6

roachprod sql local:1 -- -e "SELECT range_id, lease_holder, lease_holder_locality, replicas, replica_localities FROM [SHOW RANGES FROM TABLE fd.t WITH DETAILS];"
```

Which results in a configuration with `"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"` localities, where secondary is dead and the three remaining replicas in `tertiary` maintain quorum.

```sql

root@localhost:29016/defaultdb> select range_id, lease_holder, lease_holder_locality, replicas, replica_localities from [SHOW RANGES FROM TABLE fd.t WITH DETAILS];
range_id | lease_holder | lease_holder_locality | replicas | replica_localities
-----------+--------------+--------------------------+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------
69 | 9 | region=tertiary,zone=zone-3 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
70 | 9 | region=tertiary,zone=zone-3 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
71 | 7 | region=tertiary,zone=zone-1 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
72 | 9 | region=tertiary,zone=zone-3 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
73 | 8 | region=tertiary,zone=zone-2 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
74 | 7 | region=tertiary,zone=zone-1 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
75 | 7 | region=tertiary,zone=zone-1 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
76 | 7 | region=tertiary,zone=zone-1 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
77 | 7 | region=tertiary,zone=zone-1 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
78 | 7 | region=tertiary,zone=zone-1 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
79 | 7 | region=tertiary,zone=zone-1 | {5,6,7,8,9} | {"region=secondary,zone=zone-2","region=secondary,zone=zone-3","region=tertiary,zone=zone-1","region=tertiary,zone=zone-2","region=tertiary,zone=zone-3"}
(11 rows)

Time: 30ms total (execution 30ms / network 0ms)
```

We would expect if the constraints were "hard", in that they prohibited anything more than the quantified replica count, the range would have lost quorum after stopping nodes in the `secondary` region.

Jira issue: CRDB-42479

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.