cockroachdb / cockroachdb/cockroach
kvserver: delegated sending of committed raft log entries
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When leadership is transferred to a new replica, it does not have Raft log entries preceding its own snapshot. So it cannot send entries to a follower replica that was behind prior to the leadership transfer. This follower needs to be caught up using a snapshot. This is wasteful since other replicas have these entries (the Raft log has not been truncated to drop these entries).
With RACv2 and apply_to_all, where a send-queue can develop more commonly, this could cause unnecessary disk/cpu load on the overloaded follower to receive and apply a snapshot.
If we delegated sending of committed raft log entries, we would avoid this problem. There is no safety concern since committed entries are not overwritten.
Some other alternatives considered (suggested by @andrewbaptist):
- A new snapshot could additionally include the untruncated log: This would allow the new leader to send the entries itself. This adds overhead to every snapshot, under the pessimistic assumption that these log entries will be needed. The performance implications will likely make it hard to roll out universally
- Try avoiding lease/leadership transfer to a store that doesn't have raft entries needed to catchup a follower: This constrains the allocator further, when it is trying to load balance, which isn't desirable.
Jira issue: CRDB-44935
Contributor guide
Research direction
No files or tests are named. Start by locating the kvserver Raft leadership-transfer, snapshot-sending, and follower catch-up paths, then trace how committed log entries are selected after transfer. Done should avoid an unnecessary snapshot when another replica can provide the untruncated committed entries, with coverage for the described lagging-follower scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100