cockroachdb / cockroachdb/cockroach

kvflowcontrol: replication AC should mediate when to send a range snapshot

Open
#130,288 1 comment 0 reactions 0 assignees View on GitHub
A-replication-admission-control-v2 C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

If the replica falls behind due to store overload, sending it a snapshot can cause (a) further overload (especially wrt disk bandwidth consumption), (b) waste work (at the sender and receiver) if it is just going to fall behind again.

Replication AC has an understanding of send token availability, and could use lack of availability of send tokens to delay sending a snapshot. Say 100 ranges at node n1 have a replica at store s2, and 10 replicas are in need of a snapshot because of log truncation, and there are no (or few) send tokens available because they are getting consumed by the remaining 90 ranges, then we could choose not to send a snapshot. We would need to decide whether to wait for elastic send tokens or regular send tokens to be positive -- this can be done by computing a byte-weighted fraction of recent raft log entries on the range that are elastic or regular (wait for elastic if > 50% are elastic, and wait for regular if >= 50% are regular). Regardless of whether we wait for elastic or regular send tokens, we should probably deduct elastic send tokens to temporarily block expensive elastic work (like index backfills).

Epic [CRDB-37515](https://cockroachlabs.atlassian.net/browse/CRDB-37515)

Jira issue: CRDB-41976

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.