cockroachdb / cockroachdb/cockroach

changefeedccl: don't discard span-level checkpoint when highwater advances

Open
#148,621 2 comments 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, the span-level checkpoint is discarded whenever the highwater advances but not enough time has passed to make another span-level checkpoint (default 10m):
https://github.com/cockroachdb/cockroach/blob/f3f8b41b666d2b975c62e5d042812a4ca4eb6b1d/pkg/ccl/changefeedccl/changefeed_processors.go#L1800

This is unfortunate in the scenario when the changefeed is lagging but the highwater is still slowly making progress because it means if the changefeed restarts, we might not have a span-level checkpoint saved as highwater updates happen at a more frequent rate. As a middle ground between throwing away the checkpoint and creating a new one each time the highwater is updated, we could instead retain the existing span-level checkpoint until enough time has passed that we can make a new one.

Jira issue: CRDB-51702

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.