cockroachdb / cockroachdb/cockroach

changefeedccl: default processor distribution doesn't result in the best performance

Open
#148,100 4 comments 0 reactions 0 assignees View on GitHub
A-cdc branch-master branch-release-24.1 branch-release-24.2 branch-release-24.3 branch-release-25.1 branch-release-25.2 C-bug O-support P-3 T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Changefeeds by default use the bulk oracle, which assigns ranges randomly to nodes that have a replica for it. Changefeed processors are then put on those nodes, assigned to those ranges' spans.

While typically this results in changefeed processors running on each node in the cluster, it often results in unbalanced work across changefeed processors. The desire to colocate the work with the data is at odds with the fact that a given table's ranges' replicas are not evenly distributed across the cluster.

If you enable `changefeed.default_range_distribution_strategy="balanced_simple"`, then the value of `changefeed.balance_range_distribution.sensitivity` (default 0.05) is used to reassign replicas on nodes with more than `sensitivity` (fraction) ranges over the average, until they are no more than `sensitivity` above the average.

We should consider either setting `balanced_simple` by default, or calling it out more strongly in documentation. The tradeoff with it is it results in higher inter-node traffic.

Jira issue: CRDB-51402

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.