cockroachdb / cockroachdb/cockroach

c2c: consider destination cluster data distribution during work c2c distribution planning

Open
#99,164 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-enhancement T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When a C2C job plans work distribution, we me match each source cluster node to a destination cluster node for production and ingestion. After C2C job has done some work, what is the optimal matching of nodes?

The current policy used to distribute work across source and destination clusters may lead to a poor distribution of work as nodes cycle in either cluster. To explain, consider the current implementation:
1. On the source cluster, a call to `dsp.PartitionSpans()` maps each available node to a set of key spans, a `partition`, that it produces a stream for. `dsp.PartitionSpans` is smart enough -- it typically assigns a key span to a node if that node is the leaseholder for the underlying range, implying that each producer node is initially responsible for approximately the same amount of data/ranges/work.
2. On the destination cluster, we round robin partitions to available nodes. Given that each partition is roughly the same amount of work, we initially distribute work evenly.

Now, I think the distribution could be suboptimal if a node dies. Suppose we had this initial match between producer nodes (P*) and destination nodes (D*):
- S1 - D1
- S2 - D2
- S3 - D3
- S4 - D4
- ....
- Si-Di

Suppose that S1 dies, S2-4 acquire S1's leases, the replication job will then replan the following configuration:
- S2 - D1
- S3 - D2
- S4 - D3
- ....
- Si-Di-1

I _think_ we'd much rather have the following match:
- S2 - D2
- S3 - D3
- S4 - D4
- .....
- Si - Di

In other words, if there used to be match between two nodes, we should bias replanning to maintain that match. Why? Because D2 historically ingested all of S2's ranges' data, so D2 _likely_ contains replicas of S2's data. This bias could reduce the number of snapshots flying around the destination cluster.

The upshot of this thought is: when matching source and destination nodes during _replanning,_ we ought to match a destination node to a source node if the destination node already has a good number of replicas that overlap the source node's partition.

Jira issue: CRDB-25729

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.