cockroachdb / cockroachdb/cockroach
LDR Sizes Job Based on Source Cluster
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Logical Data Replication is implemented as a DistSQL plan that is supposed to schedule one processor per available node. Each processor is assigned a set of ranges from the source cluster and spaws goroutines to handle batches of events.
There is a bug in how LDR plans processors so that it plans one processor for each source node instead of one processor for each destination node. This can cause instability in the destination cluster. The limiting factor on [concurrency is admission controls ability to control sudden bursts of goroutines](https://github.com/cockroachdb/cockroach/issues/144877). 128 was chosen experimentally because larger values caused goroutine scheduling delays.

## Workaround
Set `logical_replication.consumer.flush_worker_per_proc` to 128 * (source_cluster_size / destination_cluster_size). For example, if the source cluster is twice as large as the destination, workers per proc should be set to 64.
Jira issue: CRDB-49587
Epic CRDB-51519
Contributor guide
Assessment
This issue has not been assessed yet.