cockroachdb / cockroachdb/cockroach

mmaprototype: add tuning knobs that SMA supports

Open
#167,778 1 comment 0 reactions 1 assignee Claimed by @wenyihu6 View on GitHub
A-kv A-kv-distribution branch-master C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

SMA's store rebalancer has several anti-thrashing knobs and tuning mechanisms. This issue catalogs each one and documents whether MMA has an equivalent, a different approach, or no coverage.

## Compile-time constants

- [ ] **`minLeaseLoadFraction`** (0.5% of store load)
- **MMA:** Identical — `minLeaseLoadFraction` (0.5%) in topK range selection ([cluster_state.go:1822](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/allocator/mmaprototype/cluster_state.go#L1822)).

- [ ] **`minReplicaLoadFraction`** (2% of store load)
- **MMA:** Identical — `minReplicaLoadFraction` (2%).

- [ ] **`MinCPUThresholdDifference`** (50ms, ~5% of 1 core)
- **MMA:** Different approach. `computeSummaryUpperBound` caps to `loadNormal` when CPU < 5% utilization. SMA uses a fixed absolute floor; MMA uses a utilization-based floor that scales with node capacity (5% of a 2-core node = 100ms, 5% of a 16-core node = 800ms).

- [ ] **`MinCPUDifferenceForTransfers`** (100ms pairwise minimum)
- **MMA:** No equivalent. SMA requires a minimum absolute CPU gap between source and target before any transfer. MMA's `canShedAndAddLoad` checks that the target's load summary doesn't worsen across dimensions, but has no absolute floor on the pairwise diff.

- [ ] **`MinQPSThresholdDifference`** (100 QPS)
- **MMA:** N/A — MMA doesn't balance QPS.

- [ ] **`maxHotRangesToLog`** (5)
- **MMA:** No direct equivalent. MMA has its own logging in `ComputeChanges`.

## Cluster settings

- [ ] **`kv.allocator.store_cpu_rebalance_threshold`** (default 0.10)
- **MMA:** Same values but hardcoded: `meanFractionSlow` = 0.10, `meanFractionNoChange` = 0.05 ([load.go:766-768](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/allocator/mmaprototype/load.go#L766)). Not tunable by operators.

- [ ] **`kv.allocator.qps_rebalance_threshold`** (default 0.10)
- **MMA:** N/A — MMA doesn't balance QPS.

- [ ] **`kv.allocator.load_based_rebalancing_interval`** (default 1 min)
- **MMA:** Same setting. MMA reuses `LoadBasedRebalanceInterval` ([mma_store_rebalancer.go:74](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/mma_store_rebalancer.go#L74)).

- [ ] **`kv.allocator.min_qps_difference_for_transfers`** (default 200 QPS)
- **MMA:** N/A — MMA doesn't balance QPS.

## IO overload handling

- [ ] **`ReplicaIOOverloadThreshold`** (0.3) + **`ReplicaIOOverloadThresholdEnforcement`** (`block_all`)
- **MMA:** Different model — uses `Disposition` system with 3 states (`OK`, `Refusing`, `Shedding`) set by the integration layer ([store_status.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/allocator/mmaprototype/store_status.go)). No operator-configurable enforcement level.

- [ ] **`LeaseIOOverloadThreshold`** (0.3) + **`LeaseIOOverloadShedThreshold`** (0.4)
- **MMA:** Different model — `LeaseDispositionRefusing` / `LeaseDispositionShedding`. Set by integration layer, not tunable thresholds.

- [ ] **`DiskUnhealthyIOOverloadScore`** (0.4)
- **MMA:** Covered — `HealthUnhealthy` → both dispositions = `Shedding` (enforced by invariant).

## Disk capacity

- [ ] **`DiskCapacityOptions`** (rebalance-to threshold, shed-and-block-all threshold)
- **MMA:** Parity — `diskUtilRefuseThreshold` (0.925), `diskUtilShedThreshold` (0.95). Reads the same cluster settings via `SetDiskUtilThresholds`.

## Behavioral mechanisms

- [ ] **`LoadBasedRebalancingMode`** (off / leases-only / leases-and-replicas)
- **MMA:** Same setting + MMA mode. Parity.

- [ ] **`objectiveProvider`** (QPS vs CPU selection)
- **MMA:** Not applicable — always multi-metric (CPU + disk + write bandwidth).

- [ ] **`FollowTheWorkloadPrefersLocal`**
- **MMA:** No equivalent. SMA avoids transferring leases away from the locality generating most requests. MMA does not consider request locality.

- [ ] **`StoreFilterSuspect`**
- **MMA:** Covered differently — `HealthUnhealthy`/`HealthDead` → disposition = `Shedding`. Replaces storepool filtering with the disposition system.

- [ ] **Pairwise transfer guard** (`MinRequiredRebalanceLoadDiff`)
- **MMA:** Different approach — `canShedAndAddLoad` is a multi-dimensional check that the target's load summary doesn't worsen, but has no absolute floor.

- [ ] **Pending change awareness**
- **MMA:** MMA is stronger — `pendingChanges` + `loadNoChange` tracks in-flight changes and suppresses further rebalancing to affected stores. SMA's store rebalancer has no pending change tracking.

- [ ] **Significance floor at low load**
- **MMA:** Covered differently — `capacityFractionForSignificanceFloor` (5% of capacity) clamps the denominator in `computeFractionAbove`. SMA uses absolute constants. Different approach, same goal.

Epic: none

Jira issue: CRDB-62637

Epic CRDB-56265

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.