cockroachdb / cockroachdb/cockroach

admission: decay qos for long-running / expensive queries

Open
#172,391 3 comments 4 reactions 0 assignees View on GitHub
A-admission-control A-kv C-bug O-agent T-admission-control T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Admission control orders waiting work strictly by priority, FIFO within a priority. This isolates QoS classes from each other, but within a class, all work is equal regardless of how much it has already consumed: a reporting query issuing its 900,000th KV request at NormalPri queues at the same priority as an OLTP point read issuing its 2nd.

Proposal

Decay the effective admission priority of a transaction as it accumulates resource consumption, so that within a QoS class, cheap/short work is implicitly prioritized over long/expensive work. @michae2 mentioned this is similar to Linux's Completely Fair Scheduler (https://en.wikipedia.org/wiki/Completely_Fair_Scheduler).

Decay signal - consumption, not wall time. Decaying on cumulative per-txn resources (KV requests admitted, bytes read, CPU attributed) rather than elapsed time seems preferable: a transaction that is long-running because it is waiting (e.g., on a lock) shouldn't be penalized further, while a fan-out read trips a consumption threshold within its first seconds. The txn's AdmissionHeader already flows to every gate, so effective priority could be derived at admission time from coordinator-tracked consumption.

Other related work:

- #163494 (credit-based flow control for kvStreamer): complementary, not overlapping — flow control bounds how much work a fan-out can have in flight; decay reorders whatever work is admitted. The escalation arguably needs both: decay
addresses the same-class queuing damage, flow control the goroutine-storm damage that priorities cannot reach (spawn and run queues are priority-blind).
- #129942 (streamer goroutine scheduling impact): the non-admission half of the same incident.

Jira issue: CRDB-65627

Contributor guide

Open the contributing guide

Research direction

Start by reading the admission-control implementation and tracing how the coordinator-tracked consumption and AdmissionHeader reach each admission gate. Review related issues #163494 and #129942 for the boundaries between admission decay, flow control, and goroutine scheduling. Done would require a decided design for the consumption signal, effective-priority calculation, and supporting tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.