cockroachdb / cockroachdb/cockroach
allocator: decommissioning a mis-replicated range with non-voters can stall
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
See this reproduction here https://github.com/cockroachdb/cockroach/commit/aaac9cff15a83a177ff645122f443a3c79e995ed.
Where the range is initially mis-replicated to have a voter violating a voter constraint, but necessary to satisfy an all replica constraint (n5 here):
```
[(n1,s1):1, (n5,s5):5, (n3,s3):3, (n7,s7):6NON_VOTER, (n9,s9):7NON_VOTER]
[0,10000): num_replicas=5 num_voters=3 constraints={'+region=a':1,'+region=b':1,'+region=c':1} voter_constraints={'+region=a'}
a,a,b,c(non-voter),d(non-voter)
b is decommissioning
topology
----
a
└── [1 2 3 4]
b
└── [5 6]
c
└── [7 8]
d
└── [9 10]
```
The range will not find a replacement because the decommissioning voter (n5, outside of `region=a`) is necessary to satisfy `constraints={'+region=a':1,'+region=b':1,'+region=c':1}`.
Details
This will hit:
https://github.com/kvoli/cockroach/blob/aaac9cff15a83a177ff645122f443a3c79e995ed/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L2103-L2105
Which fails:
https://github.com/kvoli/cockroach/blob/aaac9cff15a83a177ff645122f443a3c79e995ed/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L2009-L2009
Meaning that despite a node in region a being a no worse decommission replacement target (to at least satisfy the voter constraint), it will never get picked.
Also see slack (internal) thread [here](https://cockroachlabs.slack.com/archives/CHVV403F0/p1747079314779689).
Jira issue: CRDB-50653
Contributor guide
Assessment
This issue has not been assessed yet.