cockroachdb / cockroachdb/cockroach
kvserver: add a larger scale-up test
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We have a few tests/benchmarks that measure the speed of decommissioning and upreplication[^1]. However, we don't seem to have thorough testing of larger scale-up operations in particular in multi-store scenarios - for example, a 20 node cluster at 8 stores per node doubling in size.
This is all expected to work, and likely does work since it happens during scale testing and in prod, but it seems worth testing programmatically as well. In particular, there are some potential bottlenecks. There can (morally speaking) be at most 1 outgoing snapshot per store, and one incoming one. But each snapshot can consume up to `kv.snapshot_rebalance.max_rate` (default 32mb/s) of bandwidth, so a node with N stores can in principle consume `32*N mb/s` of bandwidth, which for large N may well exceed available network bandwidth.
[^1]: see https://roachperf.crdb.dev/?filter=replica%7Cdecom&view=replicate%2Fup%2F1to3&tab=gce
Jira issue: CRDB-52981
Contributor guide
Assessment
This issue has not been assessed yet.