cockroachdb / cockroachdb/cockroach

admission: bypassing QueryTxn/PushTxn work can starve out non-bypassing AC work

Open
#99,253 2 comments 0 reactions 0 assignees View on GitHub
A-admission-control O-support P-3 T-admission-control
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

During txn conflicts/deadlock detection, we issue PushTxns[^1] and QueryTxns[^2] as requests that bypass AC (note the missing AC headers in the references below). Bypassing work still consume AC slots/tokens[^3], so can completely starve out all other work that integrates with AC. This can contribute to amplified latencies during txn contention due to excessive AC queueing time. We saw this in an internal support case, where disabling AC got rid of the ~1s mean latencies we were adding to each request: https://github.com/cockroachlabs/support/issues/2160. There are other possible causes for AC to introduce high latencies during txn contention, especially when a large txn contends with many smaller ones.

- There could be short spikes in runnable g’s due to contending pushers, which causes AC to reduce slot count aggressively.
- AC could be too slow-reacting when increasing slot count, which we might need if suddenly requests start holding/waiting on latches longer than they were previously, due to emergent contention. So until we react, we have a lower effective slot count and are forcing a lower concurrency for non-blocked requests, forcing a lower CPU utilization. And the time taken to increase slot count is felt in latency for requests waiting for available AC slots. (https://github.com/cockroachdb/cockroach/pull/96511 will improve visibility here.)

From [internal discussion](https://cockroachlabs.atlassian.net/wiki/spaces/CORE/pages/2957901845/2023-03-22+KV+On+Call+and+Demo): Was AC making these txn deadlocks worse by (a) slowing down txns which increases the possibility of a timing-related locking deadlock, and (b) slowing down distributed deadlock detection itself?

**To Reproduce**

Run a small workload with inverted locks to run into this deadlock scenario, and see AC-bypassed QueryTxn/PushTxn work starving out foreground work.

[^1]: https://github.com/cockroachdb/cockroach/blob/71f8575f2dc0d020c850b3a0fa1047c492b5f508/pkg/kv/kvserver/intentresolver/intent_resolver.go#L408-L422

[^2]: https://github.com/cockroachdb/cockroach/blob/736a67e0d36cc545bf74d65db069ee895ff9bea0/pkg/kv/kvserver/txnwait/queue.go#L927-L937

[^3]: https://github.com/cockroachdb/cockroach/blob/71f8575f2dc0d020c850b3a0fa1047c492b5f508/pkg/util/admission/work_queue.go#L521-L531

Jira issue: CRDB-25786

Epic CRDB-25469

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.