cockroachdb / cockroachdb/cockroach
allocator: disable legacy full-disk rebalancing when MMA is enabled
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When the multi-metric allocator (MMA) is enabled, we should disable the legacy
`fullDisk` rebalancing logic in
[`rankedCandidateListForRebalancing`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L1622).
MMA has its own disk-fullness handling via the disposition system, and the
legacy logic is redundant and potentially counterproductive — it can cause
rebalancing decisions that conflict with MMA's goals. The legacy logic is
reached through two call paths (the replicate queue and the store rebalancer)
and both need to be addressed.
### Legacy full-disk rebalancing
The existing allocator actively sheds replicas off stores exceeding
`kv.allocator.max_disk_utilization_threshold` (default 95%). This lives in
[`rankedCandidateListForRebalancing`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L1622),
which marks such stores as `fullDisk: true` via
[`maxCapacityCheck`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L775).
This sets `needRebalanceFrom = true`, bypassing the normal threshold-based
rebalance check, and the `fullDisk` flag dominates candidate scoring via
[`compare()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L893)
(score of +500/-500, second only to constraint validity). Full-disk rebalances
also bypass MMA conflict checks via
[`isCriticalRebalance`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L862).
**Call path 1: Replicate queue.**
[`considerRebalance()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/plan/replicate.go#L776)
calls
[`RebalanceVoter()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go#L2064)/[`RebalanceNonVoter()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go#L2100)
→ [`RebalanceTarget()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go#L1789)
→ `rankedCandidateListForRebalancing()`, using `RangeCountScorerOptions`. When
the replicate queue processes a range whose replica lives on a full-disk store,
`fullDisk` triggers rebalancing and the allocator finds a non-full target. This
is the primary mechanism for disk-fullness shedding: the replicate queue
systematically processes every range on the store.
**Call path 2: Store rebalancer.**
[`chooseRangeToRebalance()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/store_rebalancer.go#L877)
calls
[`getRebalanceTargetsBasedOnLoad()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/store_rebalancer.go#L1098)
→ `RebalanceTarget()` → `rankedCandidateListForRebalancing()`, using
`LoadScorerOptions`. The store rebalancer fires when the **local** store is
overloaded on CPU/QPS and wants to shed load from that specific store. However,
`rankedCandidateListForRebalancing` builds equivalence classes for **all**
existing replicas of each considered range, and
[`bestRebalanceTarget`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L1937)
picks the (source, target) swap with the greatest improvement score across all
classes. Because `fullDisk` scores +500 in `compare()` — far exceeding any
load-based score — the store rebalancer can end up prioritizing removal of a
full-disk replica on a **different** store rather than removing the local
store's replica. This is unintuitive: the store rebalancer's stated goal is to
shed load from the local store, but it may instead spend its rebalancing budget
helping a different store shed disk. (The local store may still benefit
indirectly via a subsequent lease transfer to the reconfigured replica set, but
this is incidental rather than intentional.)
### MMA disk-fullness handling
MMA handles this via the disposition system in `mmaprototype/`. It uses
[`highDiskSpaceUtilization()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/mmaprototype/load.go#L687)
to detect stores exceeding thresholds, then
[`updateStoreStatuses()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/mmaprototype/cluster_state.go#L2253)
sets store dispositions to
[`ReplicaDispositionShedding`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/mmaprototype/store_status.go#L63)
(at 95% utilization) or
[`ReplicaDispositionRefusing`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/mmaprototype/store_status.go#L60)
(at 92.5%). It integrates disk pressure into multi-dimensional load balancing,
including forcing the `ByteSize` dimension in top-K selection when a store needs
to shed, and validating transfers via
[`canShedAndAddLoad()`](https://github.com/cockroachdb/cockroach/blob/a7d0270aaa2b36c5abc219c4ee01a6de3368267b/pkg/kv/kvserver/allocator/mmaprototype/cluster_state.go#L2340)
to ensure targets won't be pushed past the refuse threshold. Unlike the existing
allocator's binary filter approach, MMA tracks adjusted load (accounting for
pending changes) and uses a graduated disposition system.
Epic: CRDB-56265
Jira issue: CRDB-59549
Contributor guide
Assessment
This issue has not been assessed yet.