cockroachdb / cockroachdb/cockroach

kvserver: coalesce closed timestamp updates

Open
#107,069 2 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed A-kv-replication C-performance T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, every 200 ms `Store.startRangefeedUpdater` iterates over all replicas that have active rangefeeds, collects closed timestamps, and submits them to the replica's rangefeed processor. This wakes up O(replicas) goroutines, and emits O(replicas) checkpoint events to clients.

The goroutine overhead will be addressed by #107066. Beyond this, we should investigate whether we can reduce the overhead associated with these updates. A few considerations:

* Most ranges are idle.
* Idle ranges are tracked and updated via the closed timestamp updater, which already coalesces closed timestamp updates for all idle replicas.
* Thus most idle ranges have the same closed timestamp.

We should figure out where the greatest savings are to be had. The possible points to optimize are:

* Closed timestamp side transport bumping of replica closed timestamps: `Replica.BumpSideTransportClosed`
* Store collection of replica closed timestamps: `Store.startRangefeedUpdater` and `Replica.GetCurrentClosedTimestamp`
* Submission of closed timestamps to rangefeeds: `Store.startRangefeedUpdater` and `Replica.handleClosedTimestampUpdate`
* Processing of closed timestamps: `Processor.ForwardClosedTS` and `Processor.forwardClosedTS`
* Emission of checkpoint events: `Processor.publishCheckpoint` and `registration.publish`
* DistSender processing of checkpoint events
* kvfeed processing of checkpoint events
* ChangeFeed processing of checkpoint events

A few possible options:

* Propagate the side transport's view of idle ranges and coalesced closed timestamps all the way through the processor.
* Emit range-spanning checkpoints events across MuxRangeFeed streams.
* Make the MuxRangeFeed protocol stateful, such that individual ranges are marked as idle/active, and a single checkpoint can be emitted for all idle ranges.
* Maintain a common, low-watermark checkpoint for the entire MuxRangeFeed.
* Do nothing: the overhead here is negligible.

We should run experiments/benchmarks to determine the most cost-effective path forward.

Jira issue: CRDB-29874

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.