cockroachdb / cockroachdb/cockroach
rangefeed: reconsider errors during catch-up
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
If a rangefeed registration is disconnected during a catch-up scan, we can end up losing most of the catch-up scan progress.
Currently, errors sent to the registration via Disconnect() are sent to the stream concurrent with the catch-up scan running. The catch-up scan is then cancelled.
However, because catch-up scans only send a checkpoint at the end, this results in all of the progress of the scan being lost.
I think we can do better here. There are two paths we've discussed briefly in the past:
1. Periodically synthesize checkpoints for the portion of the keyspace we've completed the catch-up scan on. We would then also need a way to somehow use a partially-caught-up frontier when starting a rangefeed. This will reduce the amount of duplicated work to whatever was sent since the last checkpoint. I think the code to sythnesize checkpoints is probably easy. The code to use a frontier during the catch-up scan may further complicate an already complicated bit of code.
2. Queue errors until the end of the catch-up scan. Disconnect() is sent to the registration from either (1) the replica because of some range event or (2) the MuxRangeFeed call because the client disconnected. Most (all?) errors in category 1 would be perfectly valid to delivery after the catch-up scan completes and sends its checkpoint. If we placed the error in the buffer with any other events send during the catch-up scan, then we probably avoid the most common cases of losing substantial catch-up scan work because of a disconnect. We might need to distinguish errors in category (2) in our registration API since in those cases, there is no use letting the catch-up scan continue since errors in category (2) mean that no one is going to be consuming that catch-up scan anyway.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Jira issue: CRDB-56259
Contributor guide
Assessment
This issue has not been assessed yet.