cockroachdb / cockroachdb/cockroach
changefeedccl: don't discard span-level checkpoint when highwater advances
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently, the span-level checkpoint is discarded whenever the highwater advances but not enough time has passed to make another span-level checkpoint (default 10m):
https://github.com/cockroachdb/cockroach/blob/f3f8b41b666d2b975c62e5d042812a4ca4eb6b1d/pkg/ccl/changefeedccl/changefeed_processors.go#L1800
This is unfortunate in the scenario when the changefeed is lagging but the highwater is still slowly making progress because it means if the changefeed restarts, we might not have a span-level checkpoint saved as highwater updates happen at a more frequent rate. As a middle ground between throwing away the checkpoint and creating a new one each time the highwater is updated, we could instead retain the existing span-level checkpoint until enough time has passed that we can make a new one.
Jira issue: CRDB-51702
Contributor guide
Assessment
This issue has not been assessed yet.