cockroachdb / cockroachdb/cockroach

rangefeed: disjoint subscriptions amplify checkpoints

Open
#163,766 4 comments 0 reactions 0 assignees View on GitHub
A-cdc A-kv-rangefeed A-many-descriptors C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Rangefeeds publish checkpoints to every subscription every time a range's closed timestamp advances. For idle ranges, this occurs periodically, but for active ranges it occurs for whenever the oldest intent is resolved. This causes a range that is under a point transaction workload to generate O(writes) checkpoints. When a range has many tables in it, it ends up with a disjoint rangefeed subscription for every table. Each write is destine for at most one of the subscriptions, but the checkpoint ends up destine for every subscription.

On the 10k table cdc test, we observed 500K checkpoints per second for 2k writes per second. I suspect we would have seen even more checkpoints if the consumer could keep up, but the rangefeed buffers filled up and subscriptions were disconnected.

We likely need to debounce checkpoints so that we don't send hundreds of them per second for each range. Most checkpoint consumers either need a checkpoint from the oldest range they are watching (which may be idle and producing only periodic checkpoints) or they are periodically checkpointing the frontier and only need a checkpoint at some fixed interval.

Here is a heap profile showing the rangefeed buffers full of checkpoints:
Image

Here is a graph of a debug metric showing the high checkpoint rate:

Image

Jira issue: CRDB-60469

Epic CRDB-62481

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.