cockroachdb / cockroachdb/cockroach

streamingccl: add concurrency to the initial splits created by the replication target

Open
#113,953 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-enhancement O-testcluster P-3 T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In our testing cluster we noticed that on replication startup on a sufficiently large cluster the gateway node spends a significant amount of time creating initial splits and scatters. This is a necessary step but currently the splits and scatters are issued serially. We think we will see non-trivial speedup here by either running these in their own goroutine, or pushing the splits and scatters into the ingestion processor. The latter will require more state to be persisted but is an option to explore. Getting through the initial splits quicker means that we start the initial scan sooner and catchup with the primary quicker.

Jira issue: CRDB-33272

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.