cockroachdb / cockroachdb/cockroach

kvserver/allocator/mmaprototype: configurable, mean-dependent (tapering) overload threshold bands

Open
#172,000 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement O-agent T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Motivation:** #171363 traced a bulk-ingest `add_column` / `create_index`
throughput regression to excessive, low-value replica/range movement by MMA. A
significant contributor is that MMA's overload band is a flat ~10%-above-mean
(paired with a 5%-above-mean hysteresis for leaving the overloaded state),
which is far too aggressive — especially at moderate mean utilizations where
being above the mean is harmless. This issue tracks making that band a
configurable, mean-dependent (tapering) curve.

The implementation plan follows, posted verbatim.

---

# MMA Tapering Threshold Bands

Implements the design in [tapering-threshold-bands.md](tapering-threshold-bands.md): make the overload band a per-dimension piecewise-linear function of the mean, expressed in absolute native units (CPU utilization points; write-bandwidth byte-rate). There is ultimately a single classification path; when a dimension's setting is empty, a default curve is synthesized that closely (not exactly) approximates today's relative "10% of mean" formula. The relative path is removed at the end of the arc, accepting a one-off testdata diff.

## Core mechanism

Today, [load.go](pkg/kv/kvserver/allocator/mmaprototype/load.go) classifies via `fractionAbove` (relative) against three consts (`meanFractionSlow=0.1`, `meanFractionNoChange=0.05`, `meanFractionLow=-0.1`). The new absolute path computes `delta = load - mean` (utilization space for CPU, absolute-load space for write-bandwidth) and classifies against `B = band(mean)`:

- `delta > B` -> `overloadSlow`
- `B/2 <= delta <= B` -> `loadNoChange`
- `-B < delta < B/2` -> `loadNormal`
- `delta <= -B` -> `loadLow`

The `fractionUsed > 0.9` urgent override stays (capacity-known dims; never applies to write-bandwidth). The disk `<50%`-full upper bound in `computeSummaryUpperBound` stays (it is keyed on a store's *own* fullness, not the mean, so the band can't express it). The CPU `<5%` idle upper bound is **removed**: it only bites when the mean is below 5%, which is exactly where the band's low-mean value governs, so it is folded into the CPU default curve's low-mean band (band `1` near 0% mean) and is also reachable by operators via low-mean anchors. (`band >= 1` reads as "effectively never" for the utilization-bounded dims, since util is in `[0,1]`; no `inf` sentinel is needed.)

ByteSize/disk keeps all its special gates (the <50%-full upper bound, the 92.5%/95% fullness dispositions, the urgent override) and also gets a configurable band curve like the other dimensions (`kv.allocator.byte_size.overload_band`, utilization fractions). Its curve only bites in the 50%-90%-full window, but it joins the same single curve path, which is what lets us delete the relative path entirely.

## Example

The settings are per-dimension anchor lists, `mean:band,...`, interpolated linearly between anchors and clamped flat outside the anchor range. CPU and byte_size use utilization fractions (0..1); write_bandwidth uses byte-rates. A store enters overload on a dimension when its load exceeds `mean + band(mean)`.

CPU - the built-in default is roughly `kv.allocator.cpu.overload_band = '0:1,0.5:0.15,0.8:0.10'` (values to be tuned)
- mean 0% util -> band 1 -> "effectively never": at idle, nobody is flagged (this replaces the old hardcoded `<5%` idle floor)
- mean 50% util -> band 15pp -> flag a store above 65% util
- mean 65% util (midpoint) -> band ~12.5pp -> flag above ~77.5%
- mean 80% util -> band 10pp -> flag above 90%
- below 0% is n/a; between anchors interpolate linearly; above 80% the band stays 10pp
- Contrast with today's flat relative 10% (flag at `mean x 1.1`): that flags at 55% when the mean is 50% (too eager - everyone is fine) and only at 88% when the mean is 80% (too slow - the cluster is hot). The default curve is deliberately the other way around: lots of headroom when there is slack (and none at all when idle), tightening as the cluster heats up.
- `band >= 1` is the idiom for "never act here": e.g. `'0:1,0.3:0.2'` means "ignore CPU until the mean reaches 30%, then a 20pp band." (Util is in `[0,1]`, so a band of 1 can never be exceeded.)

write_bandwidth - `kv.allocator.write_bandwidth.overload_band = '0:20MiB'`
- a single anchor -> a flat 20 MiB/s headroom at every level: flag any store writing more than 20 MiB/s above the mean, regardless of absolute throughput. Absolute bands shine here, since write-bandwidth has no capacity/utilization to take a percentage of.
- a tapering variant, `'20MiB:10MiB,200MiB:20MiB'`: at a 20 MiB/s mean, flag above 30 MiB/s; at a 200 MiB/s mean, flag above 220 MiB/s - i.e. ~50% headroom when traffic is light, ~10% when it is heavy.
- a note from @tbg: write_bandwidth is the write bytes passed to the LSM, so it can undershoot real disk writes by 20-30x. 20mb/s might be too lenient, could settle for 5mb/s.

byte_size (disk fullness) - `kv.allocator.byte_size.overload_band = '0.6:0.1,0.85:0.04'`
- mean 60% full -> band 10pp -> flag a store above 70% full
- mean 85% full -> band 4pp -> flag above 89% full
- This curve only matters in the 50%-90%-full window: below 50% full a store is never shed (the upper bound), and at/above ~90% the urgent override plus the 92.5%/95% fullness dispositions take over regardless of the band.

## Plumbing

Mirror the disk-threshold precedent: parsed curves held on `clusterState` (next to `diskUtilRefuseThreshold`/`diskUtilShedThreshold`), set via a new `SetOverloadBands` on the `mmaprototype.Allocator` interface, threaded through `computeLoadSummary` -> `loadSummaryForDimension`. Settings live kvserver-side and are pushed in from [mma_store_rebalancer.go](pkg/kv/kvserver/mma_store_rebalancer.go); parsing happens via `SetOnChange` (not per-tick like the disk floats, since strings are costlier to parse). The mmaprototype package stays decoupled from `settings` (kvserver reads strings, calls `mmaprototype.ParseBandCurve`, pushes the parsed result).

Default curves (final state): there is one classification path. When a dimension's setting is empty, kvserver supplies a built-in default curve. For CPU this is a deliberately-chosen *sane* curve (not a reproduction of today); the other two stay close to today:

- CPU: roughly `0:1, 0.5:0.15, 0.8:0.10` (band `1` = "effectively never" at idle, ~15pp at 50% mean, ~10pp at 80%, clamped flat above 0.8). This bakes in low-mean idle suppression (replacing the removed `<5%` upper bound) and tapers at high mean. Exact anchor values to be tuned. It does **not** derive from `kv.allocator.store_cpu_rebalance_threshold` (that legacy setting governs the SMA store rebalancer only and no longer feeds MMA's band).
- Write-bandwidth: `band(mean) = 0.10 * max(mean, 2MiB/s)` (approximates today's hardcoded relative 0.10 + `writeBandwidthSignificanceFloor`), with a high top anchor so the flat clamp is effectively never hit in range. No universal absolute taper exists, so the default stays close to today.
- ByteSize: `band(mean) = 0.10 * max(mean, 0.05)` (approximates today; the `<50%`-full upper bound and fullness dispositions dominate anyway).

The CPU default is a real policy change (idle suppression + a wider, tapering band, motivated by the over-aggressive thrashing in #171363); the other two stay close to today. Both produce the testdata diff in commit 6, larger for CPU.

```mermaid
flowchart LR
setting["kv.allocator..overload_band (cpu | write_bandwidth | byte_size)"] -->|"non-empty: SetOnChange parse"| parse["mmaprototype.ParseBandCurve"]
setting -->|"empty: use built-in default"| synth["defaultCurve(dim) (CPU: sane idle-suppressing taper)"]
parse --> setter["Allocator.SetOverloadBands"]
synth --> setter
setter --> cs["clusterState.overloadBands"]
cs --> classify["loadSummaryForDimension (single absolute path)"]
```

## Commit arc (enshrined)

Built commit-as-you-go; each commit compiles and passes its relevant tests independently. Mechanical/doc commits precede semantic ones. The curve machinery lands with the relative path still intact and the default unchanged (commits 1-5, no behavior change); commit 6 is the single, reviewable behavioral delta: flip the empty-setting default to the synthesized curve and delete the relative path.

- Commit 1 - `mmaprototype: document the load-summary band logic` (doc-only groundwork). Expand comments in [load.go](pkg/kv/kvserver/allocator/mmaprototype/load.go) on `loadSummaryForDimension`, `computeFractionAbove`, the `meanFraction*` consts, and the significance floors: spell out the relative-band semantics and that write-bandwidth has no capacity-based overrides. No behavior change. Orients the reviewer.
- Commit 2 - `mmaprototype: add bandCurve type for tapering overload bands` (isolated new abstraction). New `band_curve.go`: `bandCurve` (sorted anchors; `band(mean) float64` via linear interpolation + flat clamp outside the range; single anchor => constant) and `ParseBandCurve(dim, string)` for the `mean:band,...` format (CPU bare fractions; write-bandwidth byte-rate suffixes via `humanizeutil`), with validation (>=1 anchor, strictly increasing means, non-negative bands). New `band_curve_test.go`. Not wired in. Red/green lives entirely in the new unit tests.
- Commit 3 - `mmaprototype: parameterize load-summary band thresholds` (mechanical, behavior-preserving). Replace the inline `meanFraction*` consts with a passed-in per-dimension descriptor that still encodes today's relative behavior; thread it through `computeLoadSummary`/`cs.computeLoadSummary`. This is the seam that lets the relative mode and the (soon-to-be-added) curve mode coexist. Existing [load_test.go](pkg/kv/kvserver/allocator/mmaprototype/load_test.go) and asim testdata unchanged prove no functional change.
- Commit 4 - `mmaprototype: add absolute curve classification mode` (semantic, additive). Extend the descriptor with an optional absolute curve; when present, classify via `delta` vs `B` as above, else use relative. Relative remains the default, so nothing changes by default. Keep urgent/idle overrides. Add datadriven cases to [load_test.go](pkg/kv/kvserver/allocator/mmaprototype/load_test.go): CPU + write-bandwidth at low/mid/high means, interpolation midpoint, clamping, urgent(>90%)/idle(<5%) overrides; plus an equivalence test asserting empty curve == relative output byte-for-byte (existing testdata unchanged).
- Commit 5 - `kvserver: wire overload-band cluster settings into MMA` (integration, no default change). Register `kv.allocator.cpu.overload_band`, `kv.allocator.write_bandwidth.overload_band`, and `kv.allocator.byte_size.overload_band` (validated string settings, default empty) near [base.go](pkg/kv/kvserver/allocator/base.go); add `SetOverloadBands` to the `Allocator` interface, `allocatorState`, and `clusterState`; parse + push from [mma_store_rebalancer.go](pkg/kv/kvserver/mma_store_rebalancer.go) (`newMMAStoreRebalancer` + `SetOnChange`). Non-empty setting selects curve mode; empty still uses the relative path, so behavior is unchanged. Pass-through in the asim rebalancer [asim/mmaintegration/mma_store_rebalancer.go](pkg/kv/kvserver/asim/mmaintegration/mma_store_rebalancer.go). Add a settings-validation test.
- Commit 6 - `mmaprototype: adopt default band curves, drop relative path` (the convenient one - the behavioral delta). Add `defaultCurve(dim)`: CPU = the sane idle-suppressing taper (`0:1, 0.5:0.15, 0.8:0.10`, values to be tuned); write-bandwidth = `0.10 * max(mean, 2MiB/s)`; ByteSize = `0.10 * max(mean, 0.05)`. Use it whenever a dimension's setting is empty (all three dims). Delete the now-dead relative classification: the `meanFraction*` consts, the relative descriptor mode, the relative branch of `computeFractionAbove`, the `*SignificanceFloor` consts, and the CPU `<5%` branch of `computeSummaryUpperBound` (folded into the CPU default's low-mean band). The disk `<50%` upper bound, fullness dispositions, and urgent override are untouched. Regenerate asim testdata under `pkg/kv/kvserver/asim/tests/testdata/` - the diff lives entirely here and is the intended behavioral improvement (notably for CPU). Replace the commit-4 equivalence test with tests asserting the new default curve's classifications at representative (mean, load) points.

## Testing

- Per-commit: `./dev test pkg/kv/kvserver/allocator/mmaprototype` (commits 1-4), focused `-f` runs for `band_curve` and `load` tests.
- Commit 5: build `pkg/kv/kvserver` and `pkg/kv/kvserver/asim/...`; run the asim datadriven tests.
- Regenerate generated code if metric/string/registry surfaces change (`./dev generate bazel` for new settings registration).
- `crlfmt -w -tab 2` on each touched `.go` file before committing.

## Open items (carried from spec)

- Write-bandwidth anchors are absolute byte-rates (cluster-specific); acceptable for the prototype, revisit if a write capacity model lands. No universal new default, so its default stays close to today.
- Hysteresis/underload ratios (0.5 / -1.0) are derived, not configurable, for now.
- ByteSize/disk gets a band-curve setting like the other dims; its taper only bites in the 50%-90%-full window, with the fullness dispositions remaining the dominant disk control. Default stays close to today.
- `band >= 1` means "effectively never over/underloaded" for the utilization-bounded dims (CPU, byte_size), since util is in `[0,1]`; this is how low-mean idle suppression is expressed (e.g. `0:1,0.3:0.2`). No `inf` sentinel needed; linear interpolation off a band of 1 also still catches extreme absolute outliers in an otherwise-idle cluster.
- The CPU default is an intentional policy change (not a reproduction of today), so commit 6 carries a deliberate asim testdata diff (larger for CPU). Exact default anchor values for all three dims to be tuned during implementation.

Contributor guide

Open the contributing guide

Research direction

Start with the design in tapering-threshold-bands.md and the existing classification in pkg/kv/kvserver/allocator/mmaprototype/load.go. Review the planned band_curve.go and band_curve_test.go, then trace integration through base.go, mma_store_rebalancer.go, and the asim rebalancer. Done means the relevant unit, integration, and asim tests pass with the configured curves and regenerated testdata.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.