cockroachdb / cockroachdb/cockroach

kv: tune single-threaded raft snapshot thoughtput

Open
#122,232 11 comments 0 reactions 0 assignees View on GitHub
A-kv-replication C-investigation C-performance T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Raft snapshots are typically throttled by the `kv.snapshot_rebalance.max_rate` setting at 32MB/s. This prevents these snapshots from being disruptive to the workload, which is a risk until snapshots are integrated into admission control (https://github.com/cockroachdb/cockroach/issues/14768).

However, there are times when users want to increase this rate to recover or rebalance more quickly, so they increase the rate limit setting. We have found that once this rate limit is increased, snapshots will still only stream at a maximum of around 120MB/s, well below the hardware limits for well-provisioned systems (10-25 GiB networks, ~1 GB/s disk IO).

We should find and eliminate the bottlenecks in snapshot transmission so that users who want to can saturate their hardware.

Jira issue: CRDB-37766

Epic CRDB-37617

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.