cockroachdb / cockroachdb/cockroach

raft: introduce persistent leader term in raft log

Open
#122,446 6 comments 0 reactions 0 assignees View on GitHub
A-kv-replication C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The raft log currently does not "remember" the last term of the leader who appended entries to the log. The [state](https://github.com/cockroachdb/cockroach/blob/31336755a5183c9eb2386c93b4a368143864bad7/pkg/raft/raftpb/raft.proto#L111-L115) of a raft instance currently contains the `Term` of its latest vote, which might or might not be the leader. This means that the content of the log is not necessarily a prefix of this `Term`'s leader.

The impact of this manifests in multiple ways:

- Safety of commit index advancement [#122100] on a follower is implicit, and relies on the leader's knowledge about the follower's log state. The leader advances commit index on a follower only as part of a `MsgApp` message, or has to [cap](https://github.com/cockroachdb/cockroach/blob/911f1ce389342459592d89fe7b78bcba3a2d265a/pkg/raft/raft.go#L643-L654) it at the follower's `Match` index.
- This may result in unnecessary delays in commit index advancement on a follower: https://github.com/etcd-io/raft/issues/138.
- The async log storage protocol implements a complex [workaround](https://github.com/cockroachdb/cockroach/blob/06c9608ec395d130c7f86d95c5b763d770b13a15/pkg/raft/rawnode.go#L277-L356) to track the in-flight entries to the storage.

We should introduce a "leader term" field into the state (both the `HardState` and the in-memory state of the raft log), with the following invariant:
```
Log.Entry[last].Term <= LeaderTerm <= Term
```

The `LeaderTerm` should be updated every time the log accepts an append from a leader. The "leader term" can be used for safety checks on the follower, before advancing the commit index. It can also be used for a simpler async log protocol.

Ultimately, the `LeaderTerm` is the missing piece of state that makes Raft log equivalent to Paxos acceptor (TODO: link to the doc). The equivalence is that the `LeaderTerm` of the log is the "max accepted proposal ID" in Paxos.

The introduction of `LeaderTerm` can be done as:
1. A start-up migration that initializes `LeaderTerm = Log.Entry[last].Term`.
2. Followed by running the code that maintains this field.

Jira issue: CRDB-37894

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.