cockroachdb / cockroachdb/cockroach
kvflowcontrol: replication AC should mediate when to send a range snapshot
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
If the replica falls behind due to store overload, sending it a snapshot can cause (a) further overload (especially wrt disk bandwidth consumption), (b) waste work (at the sender and receiver) if it is just going to fall behind again.
Replication AC has an understanding of send token availability, and could use lack of availability of send tokens to delay sending a snapshot. Say 100 ranges at node n1 have a replica at store s2, and 10 replicas are in need of a snapshot because of log truncation, and there are no (or few) send tokens available because they are getting consumed by the remaining 90 ranges, then we could choose not to send a snapshot. We would need to decide whether to wait for elastic send tokens or regular send tokens to be positive -- this can be done by computing a byte-weighted fraction of recent raft log entries on the range that are elastic or regular (wait for elastic if > 50% are elastic, and wait for regular if >= 50% are regular). Regardless of whether we wait for elastic or regular send tokens, we should probably deduct elastic send tokens to temporarily block expensive elastic work (like index backfills).
Epic [CRDB-37515](https://cockroachlabs.atlassian.net/browse/CRDB-37515)
Jira issue: CRDB-41976
Contributor guide
Assessment
This issue has not been assessed yet.