cockroachdb / cockroachdb/cockroach
kv: tune single-threaded raft snapshot thoughtput
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Raft snapshots are typically throttled by the `kv.snapshot_rebalance.max_rate` setting at 32MB/s. This prevents these snapshots from being disruptive to the workload, which is a risk until snapshots are integrated into admission control (https://github.com/cockroachdb/cockroach/issues/14768).
However, there are times when users want to increase this rate to recover or rebalance more quickly, so they increase the rate limit setting. We have found that once this rate limit is increased, snapshots will still only stream at a maximum of around 120MB/s, well below the hardware limits for well-provisioned systems (10-25 GiB networks, ~1 GB/s disk IO).
We should find and eliminate the bottlenecks in snapshot transmission so that users who want to can saturate their hardware.
Jira issue: CRDB-37766
Epic CRDB-37617
Contributor guide
Assessment
This issue has not been assessed yet.