cockroachdb / cockroachdb/cockroach

raft: make RawNode aware of log compactions

Open
#132,114 0 comments 0 reactions 1 assignee Claimed by @pav-kv View on GitHub
A-kv-replication C-enhancement
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Today, `RawNode` does not proactively discover the fact that the log has been compacted. Instead, it learns this reactively in [maybeSendAppend](https://github.com/cockroachdb/cockroach/blob/0ce38176c1c290f5d3208ad53434d4eece088230/pkg/raft/raft.go#L696-L708) when it tries to fetch entries from log storage.

We should instead notify raft about log compaction explicitly, e.g. via some `Step(MsgLogCompacted)` or `RawNode.LogCompacted(...)` method. Doing this has a few benefits.

**One**. At the point when the `RawNode` is notified about the compaction, it can discover that it can no longer catch up some followers via `MsgApp` (i.e. when `Next` is below the compaction point). So it can enter `StateSnapshot` immediately. Today, this will only happen some time later, when this node tries to send a `MsgApp` again, which in some cases (`StateProbe`, `StateReplicate` with saturated in-flight limits) will happen only in ~heartbeat timeout time.

**Two**. The `raft.Storage` [API](https://github.com/cockroachdb/cockroach/blob/0ce38176c1c290f5d3208ad53434d4eece088230/pkg/raft/storage.go#L28-L40), as well as `raft` code, have to support the `ErrCompacted` returning and handling, because `RawNode` doesn't have an up-to-date knowledge about the log state and can attempt to fetch misaligned log slices. With log compaction notifications, we can get rid of this API entirely and have `RawNode` always request log slices within the correct bounds.

**Three**. Entering the `StateSnapshot` early communicates the intent to send a snapshot to the upper layer. This approach will allow mediating snapshot sending in RACv2 more gracefully than if we implement it today.

Jira issue: CRDB-42838

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.