cockroachdb / cockroachdb/cockroach
admission: bypassing QueryTxn/PushTxn work can starve out non-bypassing AC work
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
During txn conflicts/deadlock detection, we issue PushTxns[^1] and QueryTxns[^2] as requests that bypass AC (note the missing AC headers in the references below). Bypassing work still consume AC slots/tokens[^3], so can completely starve out all other work that integrates with AC. This can contribute to amplified latencies during txn contention due to excessive AC queueing time. We saw this in an internal support case, where disabling AC got rid of the ~1s mean latencies we were adding to each request: https://github.com/cockroachlabs/support/issues/2160. There are other possible causes for AC to introduce high latencies during txn contention, especially when a large txn contends with many smaller ones.
- There could be short spikes in runnable g’s due to contending pushers, which causes AC to reduce slot count aggressively.
- AC could be too slow-reacting when increasing slot count, which we might need if suddenly requests start holding/waiting on latches longer than they were previously, due to emergent contention. So until we react, we have a lower effective slot count and are forcing a lower concurrency for non-blocked requests, forcing a lower CPU utilization. And the time taken to increase slot count is felt in latency for requests waiting for available AC slots. (https://github.com/cockroachdb/cockroach/pull/96511 will improve visibility here.)
From [internal discussion](https://cockroachlabs.atlassian.net/wiki/spaces/CORE/pages/2957901845/2023-03-22+KV+On+Call+and+Demo): Was AC making these txn deadlocks worse by (a) slowing down txns which increases the possibility of a timing-related locking deadlock, and (b) slowing down distributed deadlock detection itself?
**To Reproduce**
Run a small workload with inverted locks to run into this deadlock scenario, and see AC-bypassed QueryTxn/PushTxn work starving out foreground work.
[^1]: https://github.com/cockroachdb/cockroach/blob/71f8575f2dc0d020c850b3a0fa1047c492b5f508/pkg/kv/kvserver/intentresolver/intent_resolver.go#L408-L422
[^2]: https://github.com/cockroachdb/cockroach/blob/736a67e0d36cc545bf74d65db069ee895ff9bea0/pkg/kv/kvserver/txnwait/queue.go#L927-L937
[^3]: https://github.com/cockroachdb/cockroach/blob/71f8575f2dc0d020c850b3a0fa1047c492b5f508/pkg/util/admission/work_queue.go#L521-L531
Jira issue: CRDB-25786
Epic CRDB-25469
Contributor guide
Assessment
This issue has not been assessed yet.