cockroachdb / cockroachdb/cockroach
kvserver: reduce burstiness of bulk-low-pri rate limiter
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The recently added bulk-low-pri rate limiter (`kv.bulk_low_pri_read.max_rate`) helps reduce the average disk bandwidth consumed by jobs like TTL/BACKUP, thereby reducing their impact on foreground traffic. However, even with the limiter engaged, foreground traffic can experience elevated tail latencies (>p90 in local testing). This appears to be because the rate-limiter only deducts 1 token to admit a request and then adjusts after it's done reading (since the actual number of bytes that will need to be read is unknown at admit time), which allows bursts of requests to be admitted once the bucket refills. This burstiness is worsened by increasing the allowed concurrency (`kv.bulk_low_pri_read.max_concurrent`) since each burst contains more requests.
One potential solution for reducing burstiness would be to deduct an estimate (maybe an EWMA of the number of bytes past request read) instead of 1 token.
cc @cockroachdb/sql-queries @cockroachdb/disaster-recovery
Jira issue: CRDB-65791
Contributor guide
Research direction
Start by locating the kv.bulk_low_pri_read.max_rate and kv.bulk_low_pri_read.max_concurrent settings, then trace the rate-limiter admission and post-read token adjustment in kvserver. Reproduce the local p90 latency behavior described in the issue and inspect existing limiter tests. Done means reducing request bursts and foreground tail latency while preserving the intended average disk-bandwidth limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100