cockroachdb / cockroachdb/cockroach

kvserver/rangefeed: separate catch up buffer and raft updates buffer

Open
#125,956 0 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
Currently, catch up and raft updates buffers are both directly sent to stream. This can cause problems when catch up scans take a long time and start sending a lot of messages to stream, blocking stream.Send for too long. This blocking behaviour can cause other ranges (running normally and sending raft updates) to fall behind and also enter a catch up mode.

**Describe the solution you'd like**
We are considering a solution where catch up send and raft updates send to separate buffers. With this approach, stream.Send would first drain the raft update buffer before processing the catch up buffer to avoid causing other ranges from falling behind. Note that this solution may increase the catch up scan duration of those already lagging ranges, but we think the effect is small now and it will help prevent the spread of lag that we are currently seeing.

Update: I don't think a node level catch up buffer would actually work. We would have to use a separate catch up buffer for each stream separately. We can't just buffer all catch-up updates in one node level catch up scan buffer. Updates from catch up scans of one registration have to be sent first before sending the raft updates of the registration. And we don't want to block raft updates of other registrations from being sent because other registrations' catch up scan is slow. So we need the separation somehow.

Jira issue: CRDB-39685

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.