cockroachdb / cockroachdb/cockroach

util/admission,kv: re-visit elastic pacing during read-request evaluation

Open
#153,671 0 comments 0 reactions 0 assignees View on GitHub
A-admission-control C-enhancement T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Historically we chose not to "pace" -- i.e. pause/delay inline then continue -- during batcheval request evaluation. For example, `ExportRequest` returns early with a resume span if it runs out of elastic cpu quota, rather than pausing or delaying via a pacer. This was because in the past, we let export requests run for significant wall-time because we thought that since they were reading at a historical timestamp they would be unable to block newer foreground requests, but discovered indirect blocking was still possible, eg. if a split tried to acquire an admin latch which did block on the long running historical read-latch, and in turn blocked new writes. This led us to this return-don't-delay approach within request eval, even though delay-and-continue is our preferred approach elsewhere.

Given the high cost of returning to the client and them coming back to resume the request, we grant these requests a relatively significant quota of processing (100ms) they can do before they should take the (somewhat drastic) step of returning early. This, however, has a negative implications for the SLA for any foreground work that also wants to run in those 100ms. Ideally we'd yield to a foreground request much much sooner than 100ms given our target of <1ms scheduling latency for those requests.

Since we started dropping latches early in batch evaluation, it seems like we could consider going back to mid-evaluation delays and longer running requests without risking the priority inversion issues we had before. We should explore -- while still bounding total time in eval to some reasonable limit -- adding some amount of mid-request yielding rather than just running for 100ms unconditionally.

Internal slack thread: https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1758112041061929

Jira issue: CRDB-54546

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.