cockroachdb / cockroachdb/cockroach

LDR: Event Stream Batches Cause Goroutine Stamphead

Open
#144,877 0 comments 0 reactions 0 assignees View on GitHub
A-cross-cluster-replication A-logical-data-replication C-enhancement
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

LDR is implemented as a DistSQL plan that deploys one processor per node in the cluster. Each processor is assigned a set of source ranges and retrieves batches of replication events from the source cluster. It then splits each batch and dispatches the parts to multiple concurrent goroutines for ingestion into the local cluster.

Two settings control this behavior:
1. `logical_replication.stream_batch_size`: Sets the maximum batch flush size for the client. Default is 64 MiB.
2. `logical_replication.consumer.flush_worker_per_proc`: Sets the number of concurrent goroutines used to process each batch.

During the initial scan or when the stream lags, the client tends to produce maximum-sized batches. This results in spiky load patterns, as the processor may suddenly launch up to 128 goroutines.

This burst of goroutines is the primary reason for limiting LDR concurrency. Admission control is reactive and samples the goroutine queue depth. However, launching a large number of goroutines simultaneously reduces its effectiveness, as the system can become overloaded before admission control can respond.

To address this, the LDR processor should be restructured to stagger goroutine creation over time.

Jira issue: CRDB-49586

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.