cockroachdb / cockroachdb/cockroach

kvserver: consistency checker picks arbitrary "minority" when some/all replicas' checksums differ

Open
#173,894 1 comment 0 reactions 0 assignees View on GitHub
C-bug
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

When the consistency checker detects divergence, it groups replicas by checksum and designates the smallest group as the "minority", which is then checkpointed and terminated. If some or all replicas return distinct checksums (e.g. 3 replicas, 3 different SHAs or 5 replicas, 3 different SHAs), there could be either no majority or multiple minorities. The code acknowledges this and picks an arbitrary SHA as the minority: [`pkg/kv/kvserver/replica_consistency.go`](https://github.com/cockroachdb/cockroach/blob/8812064a015d2faf99d3fc7e15880f94042954b0/pkg/kv/kvserver/replica_consistency.go#L97)

```
// When replicas diverge, anecdotally often the minority (usually of size
// one) is in the wrong. If there's more than one smallest minority (for
// example, if three replicas all return different hashes) we pick any of
// them.
var minoritySHA string
if len(shaToIdxs) > 1 {
for sha, idxs := range shaToIdxs {
if minoritySHA == "" || len(shaToIdxs[minoritySHA]) > len(idxs) {
minoritySHA = sha
}
}
}
```

Then in `checkConsistencyImpl`, only the replicas holding `minoritySHA` are added to `args.Terminate`. Some replicas are chosen non-deterministically, checkpointed, and fatally terminated. The surviving replicas could still disagree with each other, but execution continues as if the range now has a "correct" majority.

**To Reproduce**

Theoretically: induce distinct checksum divergence on all replicas of a range (e.g. via fault injection) and let the consistency queue scan the range.

**Expected behavior**

Should be able to detect the case where there is no majority, or when there is a majority but potentially multiple minorities, and crash all the minority nodes, keeping only the agreeing majority nodes alive.

Jira issue: CRDB-67193

Contributor guide

Open the contributing guide

Research direction

Start in pkg/kv/kvserver/replica_consistency.go at the checksum grouping logic and checkConsistencyImpl. Use the described fault-injection scenario to examine cases with distinct or multiple smallest checksum groups. Done means the consistency checker detects when no agreeing majority can be established and does not continue with an arbitrary surviving group.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.