cockroachdb / cockroachdb/cockroach

kvserver: delegated sending of committed raft log entries

Open
#136,235 0 comments 0 reactions 0 assignees View on GitHub
A-kv-replication C-enhancement O-perturbation T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When leadership is transferred to a new replica, it does not have Raft log entries preceding its own snapshot. So it cannot send entries to a follower replica that was behind prior to the leadership transfer. This follower needs to be caught up using a snapshot. This is wasteful since other replicas have these entries (the Raft log has not been truncated to drop these entries).

With RACv2 and apply_to_all, where a send-queue can develop more commonly, this could cause unnecessary disk/cpu load on the overloaded follower to receive and apply a snapshot.

If we delegated sending of committed raft log entries, we would avoid this problem. There is no safety concern since committed entries are not overwritten.

Some other alternatives considered (suggested by @andrewbaptist):
- A new snapshot could additionally include the untruncated log: This would allow the new leader to send the entries itself. This adds overhead to every snapshot, under the pessimistic assumption that these log entries will be needed. The performance implications will likely make it hard to roll out universally
- Try avoiding lease/leadership transfer to a store that doesn't have raft entries needed to catchup a follower: This constrains the allocator further, when it is trying to load balance, which isn't desirable.

Jira issue: CRDB-44935

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by locating the kvserver Raft leadership-transfer, snapshot-sending, and follower catch-up paths, then trace how committed log entries are selected after transfer. Done should avoid an unnecessary snapshot when another replica can provide the untruncated committed entries, with coverage for the described lagging-follower scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.