cockroachdb / cockroachdb/cockroach
kvserver: consistency checker picks arbitrary "minority" when some/all replicas' checksums differ
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
When the consistency checker detects divergence, it groups replicas by checksum and designates the smallest group as the "minority", which is then checkpointed and terminated. If some or all replicas return distinct checksums (e.g. 3 replicas, 3 different SHAs or 5 replicas, 3 different SHAs), there could be either no majority or multiple minorities. The code acknowledges this and picks an arbitrary SHA as the minority: [`pkg/kv/kvserver/replica_consistency.go`](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/kv/kvserver/replica_consistency.go#L97)
```
// When replicas diverge, anecdotally often the minority (usually of size
// one) is in the wrong. If there's more than one smallest minority (for
// example, if three replicas all return different hashes) we pick any of
// them.
var minoritySHA string
if len(shaToIdxs) > 1 {
for sha, idxs := range shaToIdxs {
if minoritySHA == "" || len(shaToIdxs[minoritySHA]) > len(idxs) {
minoritySHA = sha
}
}
}
```
Then in `checkConsistencyImpl`, only the replicas holding `minoritySHA` are added to `args.Terminate`. Some replicas are chosen non-deterministically, checkpointed, and fatally terminated. The surviving replicas could still disagree with each other, but execution continues as if the range now has a "correct" majority.
**To Reproduce**
Theoretically: induce distinct checksum divergence on all replicas of a range (e.g. via fault injection) and let the consistency queue scan the range.
**Expected behavior**
Should be able to detect the case where there is no majority, or when there is a majority but potentially multiple minorities, and crash all the minority nodes, keeping only the agreeing majority nodes alive.
Jira issue: CRDB-67193
Contributor guide
Research direction
Start in pkg/kv/kvserver/replica_consistency.go at the checksum grouping logic and checkConsistencyImpl. Use the described fault-injection scenario to examine cases with distinct or multiple smallest checksum groups. Done means the consistency checker detects when no agreeing majority can be established and does not continue with an arbitrary surviving group.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100