cockroachdb / cockroachdb/cockroach

raft: make MsgApp.Entries in StateProbe empty

Open
#131,681 0 comments 0 reactions 0 assignees View on GitHub
A-kv-replication C-enhancement C-performance
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When the follower replication flow is in `StateProbe`, the leader will likely send `MsgApp`s [with log entries](https://github.com/cockroachdb/cockroach/blob/d0dbf6a258c0bdc66128e412f32ff6e490f33b7b/pkg/raft/tracker/progress.go#L195).

The downsides of this:

- the probes aren't subject to replication admission control, and can lead to overload
- the probes are constructed inside `RawNode` under `Replica.mu`, so `MsgApp` construction with entries may lead to IO under this mutex. We would like to minimize it: #130955.

A simple fix is to make `StateProbe` messages outright empty. However, this would have a performance impact because it's optimized for the happy case when the first probe is successful (and so some entries are replicated immediately). In a general case, this reduces post-`StateProbe` replication latency by one roundtrip to the follower.

There are some ideas in https://github.com/etcd-io/raft/issues/150 and #136296 on how to optimize these roundtrips differently. The "hints" in `MsgAppResp` can be extended to carry more information, as well as embedded in the `MsgVoteResp` to eliminate the first post-election probing roundrip.

Jira issue: CRDB-42653

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.