cockroachdb / cockroachdb/cockroach
mmaprototype: periodic spikes with mma.store.cpu.load
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
mma.store.cpu.load has been observed spiking well above actual node CPU (e.g. 250 cores on a 16-core node). We don't have enough visibility to root cause.
**Hypotheses**
1. Late timer fires inflate the rate. `recordCPUUsage` adds the absolute CPU delta to an EWMA, and GetCPUStats divides by the nominal usageRefreshInterval, not the actual elapsed time between samples. If the timer fires late (scheduling delay, GC pause), the rate is inflated. Runnable goroutines at that time was <2 according to https://cockroachlabs.slack.com/archives/C01RX2G8LT1/p1776272787859229?thread_ts=1776260791.002039&cid=C01RX2G8LT1.
2. Windowing skew. `storesCPU` is summed from per-store `Capacity().CPUPerSecond` (long-window per-replica stats, reset on lease transfer/rebalance), while `nodeUsage, sqlGatewayCPU, sqlDistCPU` are ~20s EWMAs. It might not make sense to derive the in one formula.
**Related**
- `sqlDistCPU` is fed only by `SQLCPUHandle.reportCPU` from the cancel-checker path. Elastic work cpu is not properly tracked and covered by the multiplier.
**Considered metrics to add**
1. gauges on cpu measured on the SQL side
2. gauges on the sql cpu calculated on the physical model side
3. counters on unexpected cases on the modelling side: derived values k, amp factor, errors abnormal rate etc
Jira issue: CRDB-63361
Epic CRDB-56265
Contributor guide
Assessment
This issue has not been assessed yet.