cockroachdb / cockroachdb/cockroach

kvserver: reduce burstiness of bulk-low-pri rate limiter

Open
#172,543 3 comments 0 reactions 0 assignees View on GitHub
A-kv C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The recently added bulk-low-pri rate limiter (`kv.bulk_low_pri_read.max_rate`) helps reduce the average disk bandwidth consumed by jobs like TTL/BACKUP, thereby reducing their impact on foreground traffic. However, even with the limiter engaged, foreground traffic can experience elevated tail latencies (>p90 in local testing). This appears to be because the rate-limiter only deducts 1 token to admit a request and then adjusts after it's done reading (since the actual number of bytes that will need to be read is unknown at admit time), which allows bursts of requests to be admitted once the bucket refills. This burstiness is worsened by increasing the allowed concurrency (`kv.bulk_low_pri_read.max_concurrent`) since each burst contains more requests.

One potential solution for reducing burstiness would be to deduct an estimate (maybe an EWMA of the number of bytes past request read) instead of 1 token.

cc @cockroachdb/sql-queries @cockroachdb/disaster-recovery

Jira issue: CRDB-65791

Contributor guide

Open the contributing guide

Research direction

Start by locating the kv.bulk_low_pri_read.max_rate and kv.bulk_low_pri_read.max_concurrent settings, then trace the rate-limiter admission and post-read token adjustment in kvserver. Reproduce the local p90 latency behavior described in the issue and inspect existing limiter tests. Done means reducing request bursts and foreground tail latency while preserving the intended average disk-bandwidth limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.