cockroachdb / cockroachdb/cockroach
kvserver/mmaintegration: track and evaluate MMA's CPU attribution coverage via profile tags
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Summary
The multi-metric allocator (MMA) makes rebalancing decisions from a model of
where CPU is being consumed (per-replica request/raft CPU, plus SQL gateway vs
DistSQL CPU). Today we have no programmatic way to measure how good that model
is: what fraction of the process's actual on-CPU time is accounted for
("tracked") by the subsystems MMA relies on, versus invisible to it
("untracked"). We want to be able to measure and evaluate this coverage
programmatically so we can find — and over time close — the gaps.
## Motivation
MMA's physical CPU model in
[`pkg/kv/kvserver/mmaintegration/physical_model.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/mmaintegration/physical_model.go)
decomposes node CPU into movable load (scales with KV ranges / DistSQL),
known-immovable load (SQL gateway), and an implicit "everything else" that is
folded in via amplification factors and a 3x cap. When that residual is large,
shedding cannot relieve load and MMA's decisions degrade. The residual is
exactly the CPU that no attribution subsystem accounts for, but we currently
can only reason about it indirectly (via aggregate metrics), not pinpoint what
code it is.
CPU profiles already carry rich `runtime/pprof` labels, but those labels
(`range_str`, `job`, `pebble`, ...) are derived from `logtags` and are not the
same as "this CPU was actually measured/attributed by the subsystem that feeds
MMA". For example, request CPU is measured via `grunning` around
[`Replica.MeasureReqCPUNanos`](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/replica_send.go),
and SQL CPU is measured by the
[`admission.SQLCPUHandle` / `GoroutineCPUHandle`](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/sql_cpu_handle.go)
(gateway vs DistSQL split), but neither marks the corresponding profile samples
as "tracked". So a profile cannot today tell us which samples MMA's model
actually saw.
## Proposed approach
Introduce a small CPU-profile tag taxonomy that the attribution subsystems emit
on the goroutines whose CPU they account for, and a library to evaluate
coverage from a profile.
1. **Taxonomy.** Two `pprof` label keys, each carrying a short category value:
- `mma.movable=` — CPU MMA can relocate (e.g. `sql-dist`).
- `mma.immovable=` — CPU that is intentionally/known-immovable
(e.g. `sql-gateway`). This deliberately distinguishes "accounted for but
not relocatable" from "untracked because we never instrumented it".
A sample is then movable, immovable-known, or untracked/unknown (neither
key). Untracked is the instrumentation gap.
2. **First producer: SQL CPU.** Have the `SQLCPUHandle` /
`GoroutineCPUHandle` set the label for the lifetime of each registered
goroutine (`mma.immovable=sql-gateway` at the gateway, else
`mma.movable=sql-dist`), gated on active labeled profiling
(`Settings.CPUProfileType() == CPUProfileWithLabels`) so there is no cost
off the profiling path. This is the same selective-labeling pattern already
used for `range_str`.
3. **Evaluation library.** A package under
`pkg/kv/kvserver/mmaintegration` that parses a CPU profile (via
`github.com/google/pprof/profile`) and reports CPU split across
movable / immovable-known / untracked, with a per-category breakdown, and
can write an annotated profile for viewing in `pprof`
(`-tagshow=mma.disposition`). Tested with a golden-file harness over
`testdata/*.pprof` inputs (led by a hermetic synthetic profile).
The taxonomy is general: other known-immovable consumers (compactions, GC,
backup/CDC, raft processing — which today is almost entirely unlabeled) can
later adopt the same labels to move themselves out of the "unknown" bucket.
## Scope of the initial change
- Define the `mma.movable` / `mma.immovable` label taxonomy (shared constants).
- Make the SQL CPU handle emit the labels, gated on labeled profiling.
- Add the profile-attribution evaluation library + golden-file tests.
(Plumbing detail: thread `*cluster.Settings` into the `SQLCPUProvider` so the
handle can consult `CPUProfileType()`.)
## Future work
- Adopt the taxonomy in more producers (KV request/raft CPU, compaction/GC,
background jobs) to shrink the untracked bucket.
- Wire coverage measurement into nightly tests / roachtests: capture labeled
CPU profiles during representative workloads and assert/track the
tracked-vs-untracked ratio over time, so regressions in MMA's CPU visibility
are caught automatically and improvements are quantifiable.
- Correlate the profile-derived tracked share against MMA's reported
per-store movable/immovable CPU to validate the physical model end-to-end.
## Code references
- [`pkg/kv/kvserver/mmaintegration/physical_model.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/mmaintegration/physical_model.go) — movable/immovable CPU model
- [`pkg/util/admission/sql_cpu_handle.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/admission/sql_cpu_handle.go) — SQL CPU accounting (gateway vs DistSQL)
- [`pkg/kv/kvserver/replica_send.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/replica_send.go) — request CPU measurement + `range_str` labeling
- [`pkg/util/pprofutil/labels.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/util/pprofutil/labels.go) — pprof label helpers
- [`pkg/kv/kvserver/load/node_capacity_provider.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/load/node_capacity_provider.go) — feeds SQL CPU into the model
Jira issue: CRDB-65033
Epic CRDB-66473
Contributor guide
Research direction
Read pkg/util/admission/sql_cpu_handle.go and pkg/util/pprofutil/labels.go first, then review the MMA model in pkg/kv/kvserver/mmaintegration/physical_model.go. Trace how CPUProfileType() reaches SQLCPUHandle and examine the proposed evaluation package under pkg/kv/kvserver/mmaintegration. Done means SQL CPU labels are gated to labeled profiling and golden testdata/*.pprof cases report movable, immovable-known, untracked, and per-category coverage with an annotated profile.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems, observability, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100