cockroachdb / cockroachdb/cockroach

kvserver: evaluate lower rebalance action required impact

Open
#115,111 1 comment 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement O-23.2-scale-testing O-testcluster T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
In #98889 we raised the min required rebalance impact:

```
lease rebalancing: 0.1% -> 0.5%
range rebalancing: 0.1% -> 2.0%
```

This had the benefit of reducing less impactful actions, reducing churn and likelihood of thrashing.

However, in recent scale tests we have seen when nodes have a large number of ranges which contribute less than the fractions above, persistent overload can occur.

Consider 2 groups of nodes all with equal replica/lease counts. Each replica contributes `0.1%` load but the first group has 2x the load of the second.

The first group should rebalance towards the second group. However, the first group will not because of not satisfying the min threshold.

**Describe the solution you'd like**

Evaluate a lower the threshold from 0.5/2.0% and make the setting configurable.

**Describe alternatives you've considered**

The example above is contrived, however possible. The downside of a lower threshold is less friction to moving replicas/leases which can result in thrashing when signal information on other stores is quickly overwritten and becomes stale.

Jira issue: CRDB-33888

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.