cockroachdb / cockroachdb/cockroach
admission: decay qos for long-running / expensive queries
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Admission control orders waiting work strictly by priority, FIFO within a priority. This isolates QoS classes from each other, but within a class, all work is equal regardless of how much it has already consumed: a reporting query issuing its 900,000th KV request at NormalPri queues at the same priority as an OLTP point read issuing its 2nd.
Proposal
Decay the effective admission priority of a transaction as it accumulates resource consumption, so that within a QoS class, cheap/short work is implicitly prioritized over long/expensive work. @michae2 mentioned this is similar to Linux's Completely Fair Scheduler (https://en.wikipedia.org/wiki/Completely_Fair_Scheduler).
Decay signal - consumption, not wall time. Decaying on cumulative per-txn resources (KV requests admitted, bytes read, CPU attributed) rather than elapsed time seems preferable: a transaction that is long-running because it is waiting (e.g., on a lock) shouldn't be penalized further, while a fan-out read trips a consumption threshold within its first seconds. The txn's AdmissionHeader already flows to every gate, so effective priority could be derived at admission time from coordinator-tracked consumption.
Other related work:
- #163494 (credit-based flow control for kvStreamer): complementary, not overlapping — flow control bounds how much work a fan-out can have in flight; decay reorders whatever work is admitted. The escalation arguably needs both: decay
addresses the same-class queuing damage, flow control the goroutine-storm damage that priorities cannot reach (spawn and run queues are priority-blind).
- #129942 (streamer goroutine scheduling impact): the non-admission half of the same incident.
Jira issue: CRDB-65627
Contributor guide
Research direction
Start by reading the admission-control implementation and tracing how the coordinator-tracked consumption and AdmissionHeader reach each admission gate. Review related issues #163494 and #129942 for the boundaries between admission decay, flow control, and goroutine scheduling. Done would require a decided design for the consumption signal, effective-priority calculation, and supporting tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100