cockroachdb / cockroachdb/cockroach

kvserver: improve sender-side snapshot rate limit calculation

Open
#160,106 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We've recently worked on receiver-side snapshot throttling (via disk bandwidth AC, see e.g. #159436). The sender-side throttling is becoming insufficient in this world as it computes a fixed time budget based on the snapshot rate limit cluster setting. However, snapshots can also spend time in the recipient snapshot queue (formed on top of a semaphore) and now additionally via disk bandwidth AC.

We should rework this sender-side timeout mechanism. Instead of "just" setting a context deadline, we should
- set a generous fixed deadline for the initial snapshot handshake (make sure we don't get stuck on the recipient semaphore for too long)
- then when we stream use more of a "heartbeat"-based timeout (cancel if an individual Send/Recv call takes too long).

In effect, sending shouldn't care about the particular rate, as long as there is progress. This would also allow us to eschew the sender-side rate limit entirely if store bandwidths are configured for the cluster, since the sender could simply go as fast as it could and let the receiver handle elastic throttling. Of note is that the sender then has no need to know with what bandwidth the recipient store is configured.

Epic: CRDB-58158

Jira issue: CRDB-58159

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.