cockroachdb / cockroachdb/cockroach
util/admission,kv: re-visit elastic pacing during read-request evaluation
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Historically we chose not to "pace" -- i.e. pause/delay inline then continue -- during batcheval request evaluation. For example, `ExportRequest` returns early with a resume span if it runs out of elastic cpu quota, rather than pausing or delaying via a pacer. This was because in the past, we let export requests run for significant wall-time because we thought that since they were reading at a historical timestamp they would be unable to block newer foreground requests, but discovered indirect blocking was still possible, eg. if a split tried to acquire an admin latch which did block on the long running historical read-latch, and in turn blocked new writes. This led us to this return-don't-delay approach within request eval, even though delay-and-continue is our preferred approach elsewhere.
Given the high cost of returning to the client and them coming back to resume the request, we grant these requests a relatively significant quota of processing (100ms) they can do before they should take the (somewhat drastic) step of returning early. This, however, has a negative implications for the SLA for any foreground work that also wants to run in those 100ms. Ideally we'd yield to a foreground request much much sooner than 100ms given our target of <1ms scheduling latency for those requests.
Since we started dropping latches early in batch evaluation, it seems like we could consider going back to mid-evaluation delays and longer running requests without risking the priority inversion issues we had before. We should explore -- while still bounding total time in eval to some reasonable limit -- adding some amount of mid-request yielding rather than just running for 100ms unconditionally.
Internal slack thread: https://cockroachlabs.slack.com/archives/C01SRKWGHG8/p1758112041061929
Jira issue: CRDB-54546
Contributor guide
Assessment
This issue has not been assessed yet.