cockroachdb / cockroachdb/cockroach
rangefeed: staggered start
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When starting a rangefeed, we should start the ranges that are at the replicated time and delay starting ranges that are ahead of the replicated time.
There are a few motivations for this:
1. For use cases like PCR and transactional LDR, we can't make any forward progress until the resolved time for all ranges advances past the replicated time. So starting the caught up ranges actually delays the useful data.
2. CDC users often monitor the checkpoint lag and there is a tendency in CDC for the checkpoint to stall until every range has caught up. If we started the oldest first, we would likely see incremental checkpoint progress and we caught up individual ranges.
3. The CDC blob client has a bug where partial checkpoints cause us to violate the ordering guarantee. One way to avoid this is by delaying data from spans that are more recent than the frontier's edge until the replicated time catches up to the span. https://github.com/cockroachdb/cockroach/issues/155174 https://cockroachlabs.slack.com/archives/C065X5307U3/p1774303254571759.
There is one big risk to doing this: if a range is stuck or unavailable, the rangefeed can't make any progress on start up.
Jira issue: CRDB-63284
Contributor guide
Assessment
This issue has not been assessed yet.