cockroachdb / cockroachdb/cockroach

kvserver: diversityRebalanceFromScore can be expensive

Open
#147,800 1 comment 0 reactions 0 assignees View on GitHub
A-kv C-enhancement O-25.2.1-scale-testing O-testcluster P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Periodically, we iterate over all replicas in a store and call shouldQueue. If no other actions are needed, we fall back to checking for potential rebalances via RebalanceVoter and RebalanceNonVoter. These, in turn, call rankedCandidateListForRebalancing, which iterates over all stores in the cluster to generate candidates for every existing store. For each, it computes a diversity score as if it is removing the replica from the current store and adding it to the candidate. When computing the diversity score, it performs a pairwise locality comparison across existing stores, which is an O(existing store count ^2).

There seems to be a potential optimization here where we can cache pairwise diversity score computation somehow - assuming localities of stores don't change often (https://github.com/cockroachdb/cockroach/blob/1942004afe7af1c4b786c2147544e71e947b61a6/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L2332).

Image

MMA already does optimization like this: take a look at diversityScoringMemo and existingReplicaLocalities.

---

Since MMA already has this optimization, I'm not sure if we want to work on this.

Jira issue: CRDB-51251

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.