cockroachdb / cockroachdb/cockroach
kvserver: revisit snapshot rate limits and semaphores
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The current snapshot throttling defaults are conservative and may unnecessarily slow down rebalancing and recovery operations. With disk bandwidth admission control now available, we can afford to be more aggressive and let receiver-side throttling handle contention dynamically.
## Current Settings
| Setting | Default | Description |
|---------|---------|-------------|
| `kv.snapshot_rebalance.max_rate` | 32 MiB/s | Sender-side rate limit |
| `kv.store.raft_snapshot.apply_limit` | 1 | Concurrent snapshot applies (receiver) |
| `kv.store.raft_snapshot.send_limit` | 2 | Concurrent snapshot sends (sender) |
## Proposals
### 1. Increase `kv.snapshot_rebalance.max_rate` default
**Current:** 32 MiB/s
**Proposed:** 64 MiB/s (or higher)
Network bandwidth is rarely the bottleneck. Disks typically provide >120 MiB/s throughput. The current 32 MiB/s limit is overly conservative and unnecessarily prolongs rebalancing.
If disk bandwidth admission control is enabled, consider falling forward to "unbounded" (effectively removing sender-side rate limiting) and letting receiver-side throttling handle contention. The receiver is better positioned to know its actual capacity.
### 2. Increase concurrency limits
It is less clear whether this is highly beneficial, but we could consider revisiting the concurrency limits.
**`kv.store.raft_snapshot.apply_limit`:**
Current: 1
**`kv.store.raft_snapshot.send_limit`:**
Current: 2
## Related Code
- Rate limit: `pkg/kv/kvserver/snapshot_settings.go` (`rebalanceSnapshotRate`)
- Concurrency limits: `pkg/kv/kvserver/store.go` (`snapshotApplyLimit`, `SnapshotSendLimit`)
- Receiver throttling: `pkg/kv/kvserver/store_snapshot.go` (`throttleSnapshot`)
- Queue timeout calculation: `pkg/kv/kvserver/queue.go` (`makeRateLimitedTimeoutFunc`)
## Open Questions
- What's the right heuristic to detect that disk bandwidth AC is enabled and fall forward to unbounded rate?
- Should we expose a "profile" setting (conservative/balanced/aggressive) instead of individual knobs?
- Do we need to adjust `kv.queue.process.guaranteed_time_budget` (1 min) if we increase rates/concurrency? (See also: https://github.com/cockroachdb/cockroach/issues/160106)
Jira issue: CRDB-58160
Epic CRDB-58158
Contributor guide
Assessment
This issue has not been assessed yet.