kvserver/mmaintegration: lumpy distSQL CPU produces phantom MMA store CPU load (>1400%) via uncapped amplification

**Describe the problem**

The MMA CPU physical model (`computePhysicalCPU` in `pkg/kv/kvserver/mmaintegration/physical_model.go`)
can report a per-store CPU load many times the node's actual measured CPU, with an
amplification factor far above the documented cap of `maxCPUAmplification=3`. In an
`add_column` backfill roachtest we saw `mma.store.cpu.utilization` reach **1456%** (store
load 58 cores on a 4‑vCPU node, amplification factor **87.6**) while the node's real CPU was
only ~58% (≈2.3 cores).

The trigger is a **lumpy tracked input**: the distributed‑SQL CPU rate (`SQLDistCPUNanoPerSec`)
momentarily spikes to tens of cores. But the deeper issue is a **model robustness gap**: the
model trusts its tracked sub‑components (`storesCPU`, `sqlDistCPU`, `sqlGatewayCPU`) over the
one ground‑truth, physically‑bounded measurement it has (`NodeCPURateUsage`). Any lumpy or
overcounted tracked input can therefore inflate both store load and the per‑range
amplification factor without bound. distSQL is just the input that happens to be lumpy here.

This is distinct from the node‑CPU measurement bug fixed by normalizing the node CPU EWMA by
measured elapsed time — that fix is in the analyzed build and the node CPU EWMA is clean
throughout (see Evidence). The phantom comes entirely from the distSQL input + the model.

**How it manifests**

`mma.store.cpu.*` reads wildly above physical reality for a few allocator ticks at a time:

| field | spike value | reality |
|---|---|---|
| `mma.store.cpu.utilization` | 1456% | ~58% |
| `mma.store.cpu.load` | 58.2 cores | ~2.3 cores (node total) |
| `mma.store.cpu.amplification` | 87.6 | documented cap is 3 |
| `mma.store.cpu.sql_dist` | 57.6 cores | impossible (> whole-node CPU) |
| `mma.store.cpu.replica` / `sql_gateway` | 0.67 / 0.11 cores | normal |

In this workload the spikes were rare and short (≈10 ticks of ~5600), did not move the
steady‑state balance, and did not drive the run's rebalancing churn — so the user impact in
*this* run was limited. The concern is correctness/robustness: the model can produce
arbitrarily wrong per‑store load and per‑range CPU overhead whenever a tracked CPU input is
lumpy or overcounted (e.g. a workload with large, bursty distSQL).

**Root cause**

Two separable defects:

1. **Lumpy distSQL CPU counter (source).** `SQLDistCPUNanoPerSec` is derived from the
cumulative counter `sqlCPUProviderImpl.cumulativeDistSQLCPUNanos`, which is incremented by
grunning CPU reported in lumps — periodically (`measureAndAdmit`) and as a final lump when
a handle closes (`reportCPU`). A bulk backfill runs many distSQL processors; CPU they spent
over a long real interval is reported in a burst. The node‑capacity sampler divides the
per‑poll delta by the ~1s sampling window, so a lump becomes a rate far exceeding physical
capacity. Unlike `GetProcCPUTime` (the node counter, bounded by `ncores·Δt` per interval),
this counter has no such in‑interval bound. The elapsed‑normalization that protects the
node CPU EWMA does not help, because here the *numerator* (the delta), not the divisor, is
the problem.

2. **The model amplifies lumpy inputs without bound (model).** In `computePhysicalCPU`:
```go
k := max(1, min(nodeUsage/totalTracked, maxCPUAmplification)) // only k is capped
ampFactor = (1 + sqlDistCPU/storesCPU) * k // (1 + distRatio) is NOT capped
load = storeReplicaCPU * ampFactor + immovable/numStores
immovable = max(0, nodeUsage - trackedMoveable*k) // computed from tracked sum, not from load
```
- The amplification cap is applied to `k` only; the `(1 + sqlDistCPU/storesCPU)` factor is
uncapped, so a tiny `replica` denominator turns a bad `sqlDistCPU` into a huge `ampFactor`
(here `(1 + 57.57/0.665)·1 = 87.6`).
- `load = storeReplicaCPU · ampFactor` then far exceeds `nodeUsage` (the documented
"overshoot" branch), even though `nodeUsage` is the ground‑truth bound on total node CPU.
- `ampFactor` also scales **per‑range** physical CPU (`MakePhysicalRangeLoad`), so MMA's
view of each range's CPU overhead is inflated ~30–88×, giving it badly wrong ideas about
how much load moving a single range would relieve.

**Proposed fixes**

- *Model (defense in depth; makes MMA robust to any lumpy tracked input).* Bound `ampFactor`
by two independent ceilings and re‑anchor `immovable` to the measured node CPU:
```go
ampFactor = (1 + sqlDistCPU/storesCPU) * k
// maxCPUAmplification keeps per-range physical load sane; nodeUsage/storesCPU keeps the
// moveable physical CPU (storesCPU*amp) from exceeding the node's measured CPU.
ampFactor = max(1, min(ampFactor, maxCPUAmplification, nodeUsage/storesCPU))
immovable = max(0, nodeUsage - storesCPU*ampFactor)
```
`nodeUsage` is ground truth; the tracked components should only decide how it *splits* into
moveable‑per‑range vs immovable, never inflate the total or the multiplier. This is
algebraically identical to today in the self‑consistent regime
(`trackedMoveable + sqlGatewayCPU ≤ nodeUsage`), so it is a no‑op except when a tracked input
is inconsistent with measured node CPU. (Requires updating `assertCPULoadInvariant`, whose
overshoot trigger becomes `storesCPU·ampFactor > nodeUsage`, and the "SQL‑aware
decomposition" doc comment, which currently states amp may exceed the cap by design.)
Tradeoff: when distSQL is *legitimately* > ~2× replica CPU, the excess is reclassified from
per‑range moveable into immovable rather than letting amp exceed 3 — arguably correct, since
a backfill's distSQL CPU does not move when you move one of its ranges.

- *Source (root cause; makes the metric itself trustworthy).* Make the distSQL CPU rate
reflect in‑interval work — report distSQL CPU incrementally at a bounded cadence, or
smooth/clamp at the sampler — so a single poll delta can't represent CPU spent over a much
longer prior interval.

**Evidence**

- *Node CPU EWMA is clean through the biggest spike* (n6 @ 17:17:40), from a per‑tick debug
log: every sample ~2–3 cores, `elapsed≈1.000s`; the node EWMA never exceeds 3.41 cores
across the whole backfill. So `NodeCPURateUsage` is not the source.
- *Decomposition of the spike* (per‑store `mma.store.cpu.*` gauges): only `sql_dist` spikes
(0 → 57.6 → 21.2 → 7.8 cores, an EWMA tail); `replica`, `sql_gateway`, node usage stay
normal. The amp math closes exactly: `load = replica·amp = 0.665·87.6 = 58.2`. This pattern
drives all observed spikes (stores 6/8/1/7).
- *Spikes did not drive churn:* MMA move rate was ~42/min in both spike minutes and non‑spike
minutes; ~95% of moves occurred in spike‑free minutes.

**Code references** (paths as of the analyzed build; line numbers approximate)

- [`physical_model.go` — `computePhysicalCPU` amp/immovable](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/mmaintegration/physical_model.go#L205-L271)
- [`physical_model.go` — `MakePhysicalRangeLoad` (per‑range amp)](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/mmaintegration/physical_model.go#L409-L421)
- [`node_capacity_provider.go` — `recordCPUUsage` (feeds sqlDistEWMA)](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/load/node_capacity_provider.go#L202-L253)
- [`sql_cpu_handle.go` — `reportCPU` / `cumulativeDistSQLCPUNanos`](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/sql_cpu_handle.go#L201-L617)

Jira issue: CRDB-65161

Start with computePhysicalCPU and MakePhysicalRangeLoad in pkg/kv/kvserver/mmaintegration/physical_model.go, then inspect recordCPUUsage and reportCPU in the referenced load and admission files. Compare the model's amplification and invariant behavior with NodeCPURateUsage; done means inconsistent tracked CPU cannot produce unbounded per-store or per-range load, with the related invariant and documentation updated.

Written by the indexing model from the issue text.