cockroachdb / cockroachdb/cockroach

rangefeed: reconsider errors during catch-up

Open
#156,932 0 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed branch-master C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

If a rangefeed registration is disconnected during a catch-up scan, we can end up losing most of the catch-up scan progress.

Currently, errors sent to the registration via Disconnect() are sent to the stream concurrent with the catch-up scan running. The catch-up scan is then cancelled.

However, because catch-up scans only send a checkpoint at the end, this results in all of the progress of the scan being lost.

I think we can do better here. There are two paths we've discussed briefly in the past:

1. Periodically synthesize checkpoints for the portion of the keyspace we've completed the catch-up scan on. We would then also need a way to somehow use a partially-caught-up frontier when starting a rangefeed. This will reduce the amount of duplicated work to whatever was sent since the last checkpoint. I think the code to sythnesize checkpoints is probably easy. The code to use a frontier during the catch-up scan may further complicate an already complicated bit of code.

2. Queue errors until the end of the catch-up scan. Disconnect() is sent to the registration from either (1) the replica because of some range event or (2) the MuxRangeFeed call because the client disconnected. Most (all?) errors in category 1 would be perfectly valid to delivery after the catch-up scan completes and sends its checkpoint. If we placed the error in the buffer with any other events send during the catch-up scan, then we probably avoid the most common cases of losing substantial catch-up scan work because of a disconnect. We might need to distinguish errors in category (2) in our registration API since in those cases, there is no use letting the catch-up scan continue since errors in category (2) mean that no one is going to be consuming that catch-up scan anyway.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Jira issue: CRDB-56259

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.