cockroachdb / cockroachdb/cockroach
kvserver: replace min_lease_transfer_interval with rate proportional to range count
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
`kv.allocator.min_lease_transfer_interval` controls how quickly a store can transfer away leases it owns. The default value is `500ms` and does not apply to removing a replica, draining or a transfer to satisfy lease preferences.
https://github.com/cockroachdb/cockroach/blob/3771f53b7d9d4d480d588647b91bc0988bfbc056/pkg/kv/kvserver/replicate_queue.go#L97-L100
This setting exists to prevent thrashing leases between nodes by capping the maximum change rate.
In the scale test, we noted that larger clusters are less susceptible to thrashing due to lease changes -- as there are many more ranges. This is especially true for clusters with low internode latency, where gossip updates are expected to be timely.
The setting has less benefit in these cases and is frustrating to operators waiting for lease balance post rolling-restart.
**Describe the solution you'd like**
Deprecate `kv.allocator.min_lease_transfer_interval` in favor of a setting which controls the `min_transfer_interval` in proportion to the number of ranges, e.g.
```
cluster_ranges=10k
# random value, assume this will need to be tested out
max_lease_transfer_rate_proportion=0.025
min_transfer_interval=4ms
```
**Describe alternatives you've considered**
Rate limiting could be done in proportion to the number of ranges or the number of leases. The number could come from either the local store, or the cluster (average).
Jira issue: CRDB-34127
Contributor guide
Assessment
This issue has not been assessed yet.