cockroachdb / cockroachdb/cockroach

mmaprototype: periodic spikes with mma.store.cpu.load

Open
#169,276 1 comment 0 reactions 0 assignees View on GitHub
A-kv A-kv-distribution C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

mma.store.cpu.load has been observed spiking well above actual node CPU (e.g. 250 cores on a 16-core node). We don't have enough visibility to root cause.

**Hypotheses**

1. Late timer fires inflate the rate. `recordCPUUsage` adds the absolute CPU delta to an EWMA, and GetCPUStats divides by the nominal usageRefreshInterval, not the actual elapsed time between samples. If the timer fires late (scheduling delay, GC pause), the rate is inflated. Runnable goroutines at that time was <2 according to https://cockroachlabs.slack.com/archives/C01RX2G8LT1/p1776272787859229?thread_ts=1776260791.002039&cid=C01RX2G8LT1.
2. Windowing skew. `storesCPU` is summed from per-store `Capacity().CPUPerSecond` (long-window per-replica stats, reset on lease transfer/rebalance), while `nodeUsage, sqlGatewayCPU, sqlDistCPU` are ~20s EWMAs. It might not make sense to derive the in one formula.

**Related**
- `sqlDistCPU` is fed only by `SQLCPUHandle.reportCPU` from the cancel-checker path. Elastic work cpu is not properly tracked and covered by the multiplier.

**Considered metrics to add**
1. gauges on cpu measured on the SQL side
2. gauges on the sql cpu calculated on the physical model side
3. counters on unexpected cases on the modelling side: derived values k, amp factor, errors abnormal rate etc

Jira issue: CRDB-63361

Epic CRDB-56265

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.