cockroachdb / cockroachdb/cockroach

raft: send MsgApp probes while in StateSnapshot

Open
#134,257 2 comments 0 reactions 0 assignees View on GitHub
A-kv-replication C-enhancement C-performance
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, `raft` [does not](https://github.com/cockroachdb/cockroach/blob/55cf17041236ea300c38f50bb55628d28297642f/pkg/raft/tracker/progress.go#L388) send `MsgApp` probes to a peer if its flow is in `StateSnapshot`. This stalls replication to this peer until the outstanding snapshot has been streamed.

In CRDB, snapshots can be initiated by `raft` leader, or by a leaseholder when it adds a learner replica. In leader/leaseholder split situations, there can be a leaseholder-initiated snapshot racing with the raftLeader-initiated snapshot. The raftLeader's snapshot can be queued behind the learner snapshot [[example](https://github.com/cockroachdb/cockroach/issues/133155#issuecomment-2452805188)]. This doubles the replication "stall" duration to 2x the time it takes to transfer a snapshot (e.g. in the linked example it takes ~16s). If, in the meantime, this peer is promoted to a voter (like in the example above: once the learner snapshot is done, `ChangeReplicas` promotes the replica to voter), this replication stall negatively impacts availability/latency (esp. if this voter is made the leaseholder, like in the linked example).

Another hypothetical situation like this: one leader starts streaming a snapshot to a follower, then a leader change happens, and the new leader starts streaming a snapshot too. The second snapshot is queued behind the first one, which similarly prolongs the replication stall.

To get replication unstuck, it would be beneficial for the leader to learn that the `StateSnapshot` peer actually ended up upreplicated while the snapshot was in flight. One way to achieve that: keep sending `MsgApp` probes while the flow is in `StateSnapshot`. If it happens that the peer is caught up (either by our snapshot, or by some other snapshot or delayed `MsgApp`), the leader can restore the flow to `StateReplicate` early. In the above example, the leader would restore the `MsgApp` flow 8s earlier, as soon as the leaseholder-initiated learner snapshot completes.

Downside of this approach: extra `MsgApp` probes traffic when in `StateSnapshot`. In most cases, leader == leaseholder and this race does not occur. We could try disabling this probing conditionally (e.g. if we know "locally" that there is already another snapshot in flight; or if we know that this replica is still a learner, so latency does not matter).

Jira issue: CRDB-43989

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.