cockroachdb / cockroachdb/cockroach

raft: pre-candidate can spuriously win election from vote in prior term

Open
#123,297 1 comment 0 reactions 0 assignees View on GitHub
A-kv-replication C-bug P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

From https://github.com/cockroachdb/cockroach/pull/123189#issuecomment-2083427479, see the **PreVote bug**:

> I believe that there is a bug in the existing PreVote algorithm in etcd/raft (and our new fork) that can allow a pre-vote candidate to win an pre-vote election that it otherwise shouldn't. Specifically, pre-vote elections are called with the [term following the current term](https://github.com/etcd-io/raft/blob/5a610f60542af2b7064a6bdfa6f7793a9b00c2f1/raft.go#L1037), which is carried by MsgPreVote messages. This term is also carried in [non-rejected MsgPreVoteResp messages](https://github.com/etcd-io/raft/blob/5a610f60542af2b7064a6bdfa6f7793a9b00c2f1/raft.go#L1244). Pre-vote candidates [know to handle these](https://github.com/etcd-io/raft/blob/5a610f60542af2b7064a6bdfa6f7793a9b00c2f1/raft.go#L1109) non-rejected MsgPreVoteResp messages with a future term. However, they have no special handling of non-rejected MsgPreVoteResp messages with their current term. These messages correspond to a pre-vote election held at the previous term, but it looks like they can be used to win a pre-election at the current term.
>
> For example, I think the following order of operations is allowed:
>
> ```
> 1. n3 is the leader at term 10
> 2. n1 calls a pre-vote election at term 10, sends MsgPreVote{Term: 11} to n2 and n3
> 3. n2 responds with a successful MsgPreVoteResp{Term: 11, Reject: false}, but the message is delayed
> 4. n1 never wins the pre-election
> 5. eventually the term advances to term 11 and n2 takes over as leader
> 6. n1 calls a pre-vote election at term 11, sends MsgPreVote{Term: 12} to n2 and n3
> 7. n2 and n3 both reject these, however, before they can respond...
> 8. the original message from n3 is delivered, n1 thinks it won the pre-vote election
> 9. n1 calls a disruptive election for term 12
> ```
>
> The consequences for this bug are not overly severe, because a pre-vote election is a best-effort attempt to prevent a disruptive leader election. Bugs in pre-vote do not compromise safety (i.e. leader exclusivity). Still, we should fix this.

Jira issue: CRDB-38276

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.