cockroachdb / cockroachdb/cockroach
streamingccl: add concurrency to the initial splits created by the replication target
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In our testing cluster we noticed that on replication startup on a sufficiently large cluster the gateway node spends a significant amount of time creating initial splits and scatters. This is a necessary step but currently the splits and scatters are issued serially. We think we will see non-trivial speedup here by either running these in their own goroutine, or pushing the splits and scatters into the ingestion processor. The latter will require more state to be persisted but is an option to explore. Getting through the initial splits quicker means that we start the initial scan sooner and catchup with the primary quicker.
Jira issue: CRDB-33272
Contributor guide
Assessment
This issue has not been assessed yet.