cockroachdb / cockroachdb/cockroach

LDR Sizes Job Based on Source Cluster

Open
#144,878 3 comments 0 reactions 0 assignees View on GitHub
A-cdc A-cross-cluster-replication A-disaster-recovery branch-release-25.1 C-bug P-3 T-cdc target-release-25.1.2
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Logical Data Replication is implemented as a DistSQL plan that is supposed to schedule one processor per available node. Each processor is assigned a set of ranges from the source cluster and spaws goroutines to handle batches of events.

There is a bug in how LDR plans processors so that it plans one processor for each source node instead of one processor for each destination node. This can cause instability in the destination cluster. The limiting factor on [concurrency is admission controls ability to control sudden bursts of goroutines](https://github.com/cockroachdb/cockroach/issues/144877). 128 was chosen experimentally because larger values caused goroutine scheduling delays.

![Image](https://github.com/user-attachments/assets/1b0ace1c-7977-4181-8ce7-adce14fd9ae2)

## Workaround

Set `logical_replication.consumer.flush_worker_per_proc` to 128 * (source_cluster_size / destination_cluster_size). For example, if the source cluster is twice as large as the destination, workers per proc should be set to 64.

Jira issue: CRDB-49587

Epic CRDB-51519

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.