cockroachdb / cockroachdb/cockroach

allocator: populate more decision details system.rangelog for MMA and SMA

Open
#171,464 1 comment 0 reactions 0 assignees View on GitHub
A-kv A-kv-distribution C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

`system.rangelog` is the only persistent record of rebalancing activity, but the `info.Details` field - the only per-event place to explain why a move happened - is currently either a placeholder (MMA) or too thin to support post-hoc analysis (legacy allocator). We should populate it with enough structured information that an operator can read rangelog after the fact and explain a rebalance, including diagnosing thrashing.

**Motivation**

Recent investigations have run into this gap. Example: a cluster recorded ~132k add_voter events with reason=rebalance over three hours (~5/s, flat, no convergence), spread over only ~9.8k ranges — each range moved ~13 times on average. Answering basic questions about that requires vmodules today:

- Was the driver diversity, load, IO overload, or convergence?
- Did each move actually improve the score it targeted, or did the next move revert it?
- Which store was the source, and what was its state at decision time?
- Were there equally good alternative targets, or was the choice forced?

Current state

MMA passes a placeholder: https://github.com/cockroachdb/cockroach/blob/a544dcb8eac7252700072354a17c9be1461f0ba5/pkg/kv/kvserver/mma_store_rebalancer.go#L226

We can include more details for mma decisions here:
- source / target — {store_id, locality, load: {...}} snapshots at decision time
- rejected — top-N alternative candidates with the score that ruled them out
- predicted_delta — what the allocator expected to improve (e.g. {"cpu": "-0.05"})
- allocator — mma | legacy
- time

SMA passes more details: https://github.com/cockroachdb/cockroach/blob/a544dcb8eac7252700072354a17c9be1461f0ba5/pkg/kv/kvserver/allocator/allocatorimpl/allocator.go:1580:

```
{"Target": "s5, diversity:1.00, converges:1, balance:0, rangeCount:312",
"Existing": "s7, diversity:0.83, converges:0, balance:1, rangeCount:340"}
```

This is better than MMA's placeholder. We have the check https://github.com/cockroachdb/cockroach/blob/a544dcb8eac7252700072354a17c9be1461f0ba5/pkg/kv/kvserver/allocator/allocatorimpl/allocator_scorer.go#L862-L885 here. We should include more details on the data driving this thrashing check.

Goal:
- Reconstruct source/target store state, decision at decision time.
- Identify thrashing and reasons

Epic: https://cockroachlabs.atlassian.net/browse/CRDB-56265

Jira issue: CRDB-64571

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.