cockroachdb / cockroachdb/cockroach

kvserver: replace min_lease_transfer_interval with rate proportional to range count

Open
#115,614 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement O-23.2-scale-testing O-testcluster T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

`kv.allocator.min_lease_transfer_interval` controls how quickly a store can transfer away leases it owns. The default value is `500ms` and does not apply to removing a replica, draining or a transfer to satisfy lease preferences.

https://github.com/cockroachdb/cockroach/blob/3771f53b7d9d4d480d588647b91bc0988bfbc056/pkg/kv/kvserver/replicate_queue.go#L97-L100

This setting exists to prevent thrashing leases between nodes by capping the maximum change rate.

In the scale test, we noted that larger clusters are less susceptible to thrashing due to lease changes -- as there are many more ranges. This is especially true for clusters with low internode latency, where gossip updates are expected to be timely.

The setting has less benefit in these cases and is frustrating to operators waiting for lease balance post rolling-restart.

**Describe the solution you'd like**

Deprecate `kv.allocator.min_lease_transfer_interval` in favor of a setting which controls the `min_transfer_interval` in proportion to the number of ranges, e.g.

```
cluster_ranges=10k
# random value, assume this will need to be tested out
max_lease_transfer_rate_proportion=0.025
min_transfer_interval=4ms
```

**Describe alternatives you've considered**

Rate limiting could be done in proportion to the number of ranges or the number of leases. The number could come from either the local store, or the cluster (average).

Jira issue: CRDB-34127

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.