cockroachdb / cockroachdb/cockroach
changefeedccl: default processor distribution doesn't result in the best performance
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Changefeeds by default use the bulk oracle, which assigns ranges randomly to nodes that have a replica for it. Changefeed processors are then put on those nodes, assigned to those ranges' spans.
While typically this results in changefeed processors running on each node in the cluster, it often results in unbalanced work across changefeed processors. The desire to colocate the work with the data is at odds with the fact that a given table's ranges' replicas are not evenly distributed across the cluster.
If you enable `changefeed.default_range_distribution_strategy="balanced_simple"`, then the value of `changefeed.balance_range_distribution.sensitivity` (default 0.05) is used to reassign replicas on nodes with more than `sensitivity` (fraction) ranges over the average, until they are no more than `sensitivity` above the average.
We should consider either setting `balanced_simple` by default, or calling it out more strongly in documentation. The tradeoff with it is it results in higher inter-node traffic.
Jira issue: CRDB-51402
Contributor guide
Assessment
This issue has not been assessed yet.