cockroachdb / cockroachdb/cockroach
raft: introduce persistent leader term in raft log
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The raft log currently does not "remember" the last term of the leader who appended entries to the log. The [state](https://github.com/cockroachdb/cockroach/blob/31336755a5183c9eb2386c93b4a368143864bad7/pkg/raft/raftpb/raft.proto#L111-L115) of a raft instance currently contains the `Term` of its latest vote, which might or might not be the leader. This means that the content of the log is not necessarily a prefix of this `Term`'s leader.
The impact of this manifests in multiple ways:
- Safety of commit index advancement [#122100] on a follower is implicit, and relies on the leader's knowledge about the follower's log state. The leader advances commit index on a follower only as part of a `MsgApp` message, or has to [cap](https://github.com/cockroachdb/cockroach/blob/911f1ce389342459592d89fe7b78bcba3a2d265a/pkg/raft/raft.go#L643-L654) it at the follower's `Match` index.
- This may result in unnecessary delays in commit index advancement on a follower: https://github.com/etcd-io/raft/issues/138.
- The async log storage protocol implements a complex [workaround](https://github.com/cockroachdb/cockroach/blob/06c9608ec395d130c7f86d95c5b763d770b13a15/pkg/raft/rawnode.go#L277-L356) to track the in-flight entries to the storage.
We should introduce a "leader term" field into the state (both the `HardState` and the in-memory state of the raft log), with the following invariant:
```
Log.Entry[last].Term <= LeaderTerm <= Term
```
The `LeaderTerm` should be updated every time the log accepts an append from a leader. The "leader term" can be used for safety checks on the follower, before advancing the commit index. It can also be used for a simpler async log protocol.
Ultimately, the `LeaderTerm` is the missing piece of state that makes Raft log equivalent to Paxos acceptor (TODO: link to the doc). The equivalence is that the `LeaderTerm` of the log is the "max accepted proposal ID" in Paxos.
The introduction of `LeaderTerm` can be done as:
1. A start-up migration that initializes `LeaderTerm = Log.Entry[last].Term`.
2. Followed by running the code that maintains this field.
Jira issue: CRDB-37894
Contributor guide
Assessment
This issue has not been assessed yet.