cockroachdb / cockroachdb/cockroach

kvserver: add a larger scale-up test

Open
#150,898 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We have a few tests/benchmarks that measure the speed of decommissioning and upreplication[^1]. However, we don't seem to have thorough testing of larger scale-up operations in particular in multi-store scenarios - for example, a 20 node cluster at 8 stores per node doubling in size.

This is all expected to work, and likely does work since it happens during scale testing and in prod, but it seems worth testing programmatically as well. In particular, there are some potential bottlenecks. There can (morally speaking) be at most 1 outgoing snapshot per store, and one incoming one. But each snapshot can consume up to `kv.snapshot_rebalance.max_rate` (default 32mb/s) of bandwidth, so a node with N stores can in principle consume `32*N mb/s` of bandwidth, which for large N may well exceed available network bandwidth.

[^1]: see https://roachperf.crdb.dev/?filter=replica%7Cdecom&view=replicate%2Fup%2F1to3&tab=gce

Jira issue: CRDB-52981

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.