cockroachdb / cockroachdb/cockroach

allocator: shed leases on extended CPU overload relative cluster mean

Open
#127,975 1 comment 0 reactions 0 assignees View on GitHub
A-kv A-kv-distribution C-enhancement O-testcluster P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
Its rare but not impossible for a subsystem within CRDB to misbehave or behave unexpectedly, pegging process CPU utilization near 100%, while the rest of the cluster has vacant capacity.

CPU rebalancing attributes kvserver CPU usage to replicas and in most situations is sufficient, as the CPU usage is acted upon by transferring leases anyway. In cases where its not attributed, CPU rebalancing is insufficient.

**Describe the solution you'd like**

Add functionality to the allocator which will shed all leases from a node when its

1. CPU utilization is pegged >90% for an extended duration **AND**
2. CPU utilization is more than 1.5x the mean cluster CPU utilization.

**Describe alternatives you've considered**

Using process CPU utilization, instead of aggregate replica CPU usage, this problem is also solved via balancing (there's still a risk of no leases or replicas to transfer away).

Jira issue: CRDB-40714

Epic CRDB-56265

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.