cockroachdb / cockroachdb/cockroach

allocator: decommissioning a mis-replicated range with non-voters can stall

Open
#146,681 3 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution branch-master C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

See this reproduction here https://github.com/cockroachdb/cockroach/commit/aaac9cff15a83a177ff645122f443a3c79e995ed.

Where the range is initially mis-replicated to have a voter violating a voter constraint, but necessary to satisfy an all replica constraint (n5 here):

```
[(n1,s1):1, (n5,s5):5, (n3,s3):3, (n7,s7):6NON_VOTER, (n9,s9):7NON_VOTER]

[0,10000): num_replicas=5 num_voters=3 constraints={'+region=a':1,'+region=b':1,'+region=c':1} voter_constraints={'+region=a'}

a,a,b,c(non-voter),d(non-voter)
b is decommissioning

topology
----
a
└── [1 2 3 4]
b
└── [5 6]
c
└── [7 8]
d
└── [9 10]
```

The range will not find a replacement because the decommissioning voter (n5, outside of `region=a`) is necessary to satisfy `constraints={'+region=a':1,'+region=b':1,'+region=c':1}`.

Details

This will hit:

https://github.com/kvoli/cockroach/blob/aaac9cff15a83a177ff645122f443a3c79e995ed/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L2103-L2105

Which fails:

https://github.com/kvoli/cockroach/blob/aaac9cff15a83a177ff645122f443a3c79e995ed/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L2009-L2009

Meaning that despite a node in region a being a no worse decommission replacement target (to at least satisfy the voter constraint), it will never get picked.


Also see slack (internal) thread [here](https://cockroachlabs.slack.com/archives/CHVV403F0/p1747079314779689).

Jira issue: CRDB-50653

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.