cockroachdb / cockroachdb/cockroach
tenantrate: rate limiting can starve critical internal traffic
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Multi-tenant external process sql server deployments of CRDB use a [KV side tenant rate limiter](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/kvserver/tenantrate/limiter.go) to control how much of a single kv server a tenant can consume. The rate limiter is distinct from admission control because it tries to limit a tenant to a percentage of node capacity even if there is slack capacity. Admission control on the other hand will only kick in if the node is observing side effects of overload like increased go routine queue depth or LSM inversion.
If a tenant is running into the kv side rate limits, it can starve critical traffic like sql liveness and sql leases. Starving sql liveness causes sql servers to crash with the following error log: "exiting heartbeat loop with error: session record deleted". This is not the only cause of the "exiting heartbeat loop" error. The error occurs whenever a sql server is unable to communicate with the kv layer for more than 30 seconds.
Critical internal traffic should be exempt from tenantrate limiter.
Jira issue: CRDB-33986
gz#21258
Contributor guide
Assessment
This issue has not been assessed yet.