cockroachdb / cockroachdb/cockroach

allocator: take lease preferences into account for lease convergence

Open
#93,258 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When load-based lease rebalancing is disabled, the allocator falls back to rebalancing leases by count. It does this by calculating the mean lease count across the cluster, and then transfer a lease from a node with `leases > mean` to one with `leases < mean`. However, this does not take lease preferences into account, and the mean calculation will include the 0 lease counts from nodes that are not within the lease preferences.

Consider an example with three nodes in each of `asia`, `eu`, `us`, and a lease preference set to `us`:

* `asia1`: 0
* `asia2`: 0
* `asia3`: 0
* `eu1`: 0
* `eu2`: 0
* `eu3`: 0
* `us1`: 133
* `us2`: 133
* `us3`: 33

In this case, the operator might expect the leases to rebalance such that all `us` nodes have ~100 leases each, but this won't happen. `us3` will be stuck at 33 leases, because the average lease count is 300 / 9 = 33 and it's therefore at the mean and thus not considered underfull.

The lease convergence calculation needs to take lease preferences into account, as well as which preferences can actually be satisfied (e.g. in the case of unavailable nodes/regions).

This determination is made here:

https://github.com/cockroachdb/cockroach/blob/d4a2c743e02c82a903fdbac782f1d5df5e8c33a1/pkg/kv/kvserver/allocator.go#L2170-L2186

Based on the lease candidate store mean which is calculated here:

https://github.com/cockroachdb/cockroach/blob/0f816899cd3e8605b3d8955967a8956b4cf15fb9/pkg/kv/kvserver/store_pool.go#L778

Jira issue: CRDB-22249

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.