cockroachdb / cockroachdb/cockroach

Application Clusters Can Starve the System Cluster

Open
#119,417 4 comments 0 reactions 0 assignees View on GitHub
A-admission-control C-bug T-serverless
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

[Admission control is managed via several distinct queues](https://github.com/cockroachdb/cockroach/blob/master/docs/tech-notes/admission_control.md):

KV: work admitted at the KV layer
KV-SQL: response transferred from the KV layer to the SQL layer
SQL-SQL: response transferred from a distsql leaf to a distsql root

When the system is overloaded, tokens are allocated with preference to the lowest level queue. I.E. if the KV queue contains pending work, no tokens will be distributed to the KV-SQL or SQL-SQL queues. This works because starving the SQL queues will eventually reduce the submission rate to the KV queue, which will allow the system to distribute tokens to the SQL queues.

In an external process deployment of CRDB (e.g. serverless), the KV-SQL and SQL-SQL queues live inside the external process SQL server. Which means from the perspective of the system cluster, tenants only submit traffic to the KV queue. This allows tenants to starve the system KV-SQL and SQL-SQL queues, because there is no back pressure to prevent them from submitting work to the KV queue.

**To Reproduce**

1. Create a roachprod cluster with at least one external process application cluster.
2. Run a workload on the application cluster that generates a large amount of cpu admission control queuing at the KV layer. An easy way to achieve this is a write only KV workload with batching.
3. Pick one of the overloaded KV servers and attempt to connect to the system cluster using a SQL shell. The SQL shell will hang since authorization depends on the KV-SQL queue.

**What was the impact?**

When a physical cluster is overloaded by an application virtual cluster, the system cluster is unable to process any SQL. It's unclear what the exact impact is, but here are a few user facing consequences:

1. Tenant RU accounting stalls.
2. Tenant creation fails.
3. Backups stall.

**Workarounds**

External process deployments can work around this by disable the KV-SQL and SQL-SQL queues for the system virtual cluster.

Jira issue: CRDB-36184

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.