cockroachdb / cockroachdb/cockroach
allocator: populate more decision details system.rangelog for MMA and SMA
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
`system.rangelog` is the only persistent record of rebalancing activity, but the `info.Details` field - the only per-event place to explain why a move happened - is currently either a placeholder (MMA) or too thin to support post-hoc analysis (legacy allocator). We should populate it with enough structured information that an operator can read rangelog after the fact and explain a rebalance, including diagnosing thrashing.
**Motivation**
Recent investigations have run into this gap. Example: a cluster recorded ~132k add_voter events with reason=rebalance over three hours (~5/s, flat, no convergence), spread over only ~9.8k ranges — each range moved ~13 times on average. Answering basic questions about that requires vmodules today:
- Was the driver diversity, load, IO overload, or convergence?
- Did each move actually improve the score it targeted, or did the next move revert it?
- Which store was the source, and what was its state at decision time?
- Were there equally good alternative targets, or was the choice forced?
Current state
MMA passes a placeholder: https://github.com/cockroachdb/cockroach/blob/a544dcb8eac7252700072354a17c9be1461f0ba5/pkg/kv/kvserver/mma_store_rebalancer.go#L226
We can include more details for mma decisions here:
- source / target — {store_id, locality, load: {...}} snapshots at decision time
- rejected — top-N alternative candidates with the score that ruled them out
- predicted_delta — what the allocator expected to improve (e.g. {"cpu": "-0.05"})
- allocator — mma | legacy
- time
SMA passes more details: https://github.com/cockroachdb/cockroach/blob/a544dcb8eac7252700072354a17c9be1461f0ba5/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go:1580:
```
{"Target": "s5, diversity:1.00, converges:1, balance:0, rangeCount:312",
"Existing": "s7, diversity:0.83, converges:0, balance:1, rangeCount:340"}
```
This is better than MMA's placeholder. We have the check https://github.com/cockroachdb/cockroach/blob/a544dcb8eac7252700072354a17c9be1461f0ba5/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L862-L885 here. We should include more details on the data driving this thrashing check.
Goal:
- Reconstruct source/target store state, decision at decision time.
- Identify thrashing and reasons
Epic: https://cockroachlabs.atlassian.net/browse/CRDB-56265
Jira issue: CRDB-64571
Contributor guide
Assessment
This issue has not been assessed yet.