cockroachdb / cockroachdb/cockroach
LDR: Event Stream Batches Cause Goroutine Stamphead
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
LDR is implemented as a DistSQL plan that deploys one processor per node in the cluster. Each processor is assigned a set of source ranges and retrieves batches of replication events from the source cluster. It then splits each batch and dispatches the parts to multiple concurrent goroutines for ingestion into the local cluster.
Two settings control this behavior:
1. `logical_replication.stream_batch_size`: Sets the maximum batch flush size for the client. Default is 64 MiB.
2. `logical_replication.consumer.flush_worker_per_proc`: Sets the number of concurrent goroutines used to process each batch.
During the initial scan or when the stream lags, the client tends to produce maximum-sized batches. This results in spiky load patterns, as the processor may suddenly launch up to 128 goroutines.
This burst of goroutines is the primary reason for limiting LDR concurrency. Admission control is reactive and samples the goroutine queue depth. However, launching a large number of goroutines simultaneously reduces its effectiveness, as the system can become overloaded before admission control can respond.
To address this, the LDR processor should be restructured to stagger goroutine creation over time.
Jira issue: CRDB-49586
Contributor guide
Assessment
This issue has not been assessed yet.