cockroachdb / cockroachdb/cockroach

kvserver: revisit snapshot rate limits and semaphores

Open
#160,107 1 comment 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The current snapshot throttling defaults are conservative and may unnecessarily slow down rebalancing and recovery operations. With disk bandwidth admission control now available, we can afford to be more aggressive and let receiver-side throttling handle contention dynamically.

## Current Settings

| Setting | Default | Description |
|---------|---------|-------------|
| `kv.snapshot_rebalance.max_rate` | 32 MiB/s | Sender-side rate limit |
| `kv.store.raft_snapshot.apply_limit` | 1 | Concurrent snapshot applies (receiver) |
| `kv.store.raft_snapshot.send_limit` | 2 | Concurrent snapshot sends (sender) |

## Proposals

### 1. Increase `kv.snapshot_rebalance.max_rate` default

**Current:** 32 MiB/s
**Proposed:** 64 MiB/s (or higher)

Network bandwidth is rarely the bottleneck. Disks typically provide >120 MiB/s throughput. The current 32 MiB/s limit is overly conservative and unnecessarily prolongs rebalancing.

If disk bandwidth admission control is enabled, consider falling forward to "unbounded" (effectively removing sender-side rate limiting) and letting receiver-side throttling handle contention. The receiver is better positioned to know its actual capacity.

### 2. Increase concurrency limits

It is less clear whether this is highly beneficial, but we could consider revisiting the concurrency limits.

**`kv.store.raft_snapshot.apply_limit`:**
Current: 1

**`kv.store.raft_snapshot.send_limit`:**
Current: 2

## Related Code

- Rate limit: `pkg/kv/kvserver/snapshot_settings.go` (`rebalanceSnapshotRate`)
- Concurrency limits: `pkg/kv/kvserver/store.go` (`snapshotApplyLimit`, `SnapshotSendLimit`)
- Receiver throttling: `pkg/kv/kvserver/store_snapshot.go` (`throttleSnapshot`)
- Queue timeout calculation: `pkg/kv/kvserver/queue.go` (`makeRateLimitedTimeoutFunc`)

## Open Questions

- What's the right heuristic to detect that disk bandwidth AC is enabled and fall forward to unbounded rate?
- Should we expose a "profile" setting (conservative/balanced/aggressive) instead of individual knobs?
- Do we need to adjust `kv.queue.process.guaranteed_time_budget` (1 min) if we increase rates/concurrency? (See also: https://github.com/cockroachdb/cockroach/issues/160106)

Jira issue: CRDB-58160

Epic CRDB-58158

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.