cockroachdb / cockroachdb/cockroach

mma: high cpu usage for tryConstructMMARangeMsg

Open
#169,290 1 comment 0 reactions 0 assignees View on GitHub
A-kv A-kv-distribution branch-master C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Taken from telemetry cluster: tryConstructMMARangeMsg is consuming high cpu. We should see if there are any low hanging fruit optimization we could do. There was a TODO we had in our [spreadsheet](https://docs.google.com/spreadsheets/d/1YGZFZ0r2jTbadzwcb8pgx0MOE0ZREegZRC8z9gczV_c/edit?gid=1408217171#gid=1408217171) on this related land - kvserver: check if r.leaseStatusAtRLocked is necessary.

From claude, possible improvements:

Cheap wins (mechanical, large payoff)

1. Switch raftStatusRLocked() → raftSparseStatusRLocked() in tryConstructMMARangeMsg.
raft.RawNode.Status() deep-copies Config and every entry's Inflights. None of those are read by the consumer (raftutil.ReplicaIsBehind only touches RaftState, Lead, Commit, Progress[id].State/Match). Sparse status returns the same
observable info without the clones. This is what the bottom of the flame graph (Status → getProgress → Visit → Clone) is showing.
Requires either a ReplicaIsBehindSparse variant or widening ReplicaIsBehind to take an interface that both *Status and *SparseStatus satisfy. (mma_replica_store.go:111, replica.go:1888, raftutil/util.go:20.)
2. Stop allocating a fresh []StoreIDAndReplicaState per range per tick.
constructRangeMsgReplicas does make([]…, 0, len(desc.InternalReplicas)) for every leaseholder replica every rebalancer tick. The TODO at mma_replica_store.go:126 already calls this out — pass scratch memory in from MakeStoreLeaseholderMsg,
reset and reuse per call. With thousands of leaseholders this is constant allocator pressure.
3. Stop boxing raft.Status to a heap allocation.
raftStatusRLocked returns *raft.Status only because of how it's currently used. The TODO at replica.go:1886 calls this out.
4. Reuse the outer []RangeMsg slice across ticks in MakeStoreLeaseholderMsg.

Medium-effort, structural

5. Skip non-leaseholder replicas before taking r.mu.RLock and computing lease/raft status.
tryConstructMMARangeMsg currently locks, computes leaseStatusAt, and then discovers it isn't the leaseholder. On a node with many follower replicas this is wasted work per tick. The cheap r.shMu.state.Lease.OwnedBy(...) fast path mentioned
in the existing TODO at mma_replica_store.go:102-106 is the right shape — read it without the lock as a hint, and only fall through to the full path when it looks like we are the leaseholder. The "stale lease" tradeoff in the TODO is fine
here because the next tick corrects it.
6. Lift r.LoadStats() and r.GetMVCCStats() into a single locked region.
mmaRangeLoad calls both separately; each takes its own lock internally. Folding the reads into the same RLock as isLeaseholderWithDescAndConfig (or returning all the fields together) cuts mutex traffic per replica in half. Worth checking
that the existing methods don't do significant work outside the lock.
7. Cache amp := ms.amplificationFactors() per tick (already done at mma_replica_store.go:267) — but verify it's not recomputed inside mmaRangeLoad's callees.
Looks correct as-is; flagging only because amp was singled out as a hoist target.

Image

Jira issue: CRDB-63366

Epic CRDB-56265

Contributor guide

Open the contributing guide

Research direction

Start at tryConstructMMARangeMsg in mma_replica_store.go and inspect the TODOs around lines 102-126, then trace ReplicaIsBehind through replica.go and raftutil/util.go. Use the telemetry and flame graph as the baseline, and verify that any selected optimization preserves range-message behavior while reducing CPU or allocation overhead.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.