cockroachdb / cockroachdb/cockroach

kvserver: add knob that forces node to shed leases and be unable to re-acquire

Open
#116,061 1 comment 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

There are scenarios where a node should not hold leases due to impact on the workload, primarily when the node is overloaded or operating on faulty hardware.

In these scenarios, automatic mechanisms may not reliably kick in and having a the ability to force leases off the node without draining it is desirable.

**Describe the solution you'd like**

Introduce a knob (`crdb_internal` or CLI), which marks the node as unable to hold a lease and also kicks off shedding leases from the node.

Note relying on the replicate queue to shed leases won't reliably work, as other high priority actions (up-replication) will get acted upon first. The solution should either piggyback off the range lease drain phase, or introduce another mechanism.

**Describe alternatives you've considered**

Setting an anti-affinity for the node using lease preferences could be a possible solution, however runs into the same problem as above, stuck behind other replicate queue actions.

**Additional context**

[(internal) discussion.](https://cockroachlabs.slack.com/archives/C0KB9Q03D/p1702137452796939)

Jira issue: CRDB-34424

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.