cockroachdb / cockroachdb/cockroach

kvserver: improve recovery speed of underreplicated ranges after replacing a store

Open
#108,561 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
Is a store is replaced without first decommissioning, all the upreplication traffic will be targeted to the new store. The incoming snapshot rate on that store will be the limit to how fast it can receive traffic as it is only allows a single replica at a time.

Ideally, the upreplication would happen against all the stores in the target region, then after the system is healthy, the data could trickle back to the new store. The system needs to balance between three tradeoffs:
1) Fast upreplication
2) Overloading of a store which impacts performance
3) Unnecessary network traffic and cost by moving data twice

**Describe the solution you'd like**
The solution would be to compute a upreplication speedup ratio. The upreplicationup speed ratio is computed as "expected difference between using good vs best allocator". If there is only 1 possible node as a target, then the value is 1, if there are 9 possible nodes with good vs best, then the value is 9. This is configured with a default value of 3.0 which means that it will choose a worse allocator target (which will then require a second move) if the number of possible good targets is 3x the number of best targets. Making this value 1.0 will exactly mimic the "good" allocator today. Making this value "infinity" will exactly mimic the current behavior.

**Describe alternatives you've considered**
The alternatives are to use the good allocator like #86265. This

**Additional context**
See this internal [slack thread](https://cockroachlabs.slack.com/archives/C05L0PLK1T2/p1690579048238779) for discussion.

Jira issue: CRDB-30520

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.