cockroachdb / cockroachdb/cockroach

tenantrate: rate limiting can starve critical internal traffic

Open
#115,251 3 comments 0 reactions 0 assignees View on GitHub
A-multitenancy C-bug P-3 T-serverless
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Multi-tenant external process sql server deployments of CRDB use a [KV side tenant rate limiter](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/tenantrate/limiter.go) to control how much of a single kv server a tenant can consume. The rate limiter is distinct from admission control because it tries to limit a tenant to a percentage of node capacity even if there is slack capacity. Admission control on the other hand will only kick in if the node is observing side effects of overload like increased go routine queue depth or LSM inversion.

If a tenant is running into the kv side rate limits, it can starve critical traffic like sql liveness and sql leases. Starving sql liveness causes sql servers to crash with the following error log: "exiting heartbeat loop with error: session record deleted". This is not the only cause of the "exiting heartbeat loop" error. The error occurs whenever a sql server is unable to communicate with the kv layer for more than 30 seconds.

Critical internal traffic should be exempt from tenantrate limiter.

Jira issue: CRDB-33986

gz#21258

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.