cockroachdb / cockroachdb/cockroach
How to rebalance prior to decommissioning a dead node?
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Attempting to decommission a drained node results in,
```
[root@cockroachdb-client-secure-[snip] cockroach]# ./cockroach node decommission 3 --host cockroachdb-public --certs-dir /cockroach-certs/node
id | is_live | replicas | is_decommissioning | membership | is_draining | readiness | blocking_ranges
-----+---------+----------+--------------------+------------+-------------+-------------------+------------------
3 | false | 58 | false | active | true | allocation errors | 57
(1 row)
ranges blocking decommission detected
n3 has 1 replicas blocked with error: "range r43 requires unexpected allocation action: consider rebalance"
n3 has 1 replicas blocked with error: "range r2 requires unexpected allocation action: consider rebalance"
n3 has 1 replicas blocked with error: "range r5 requires unexpected allocation action: consider rebalance"
n3 has 1 replicas blocked with error: "range r6 requires unexpected allocation action: consider rebalance"
n3 has 1 replicas blocked with error: "range r17 requires unexpected allocation action: consider rebalance"
...more blocking errors detected.
ERROR: Cannot decommission nodes.
Failed running "node decommission"
```
But I have no idea (and searching the documentation has not helped me) as to how to "consider rebalance".
I end up waiting for the rest of the cluster to declare the node dead, at which point it happen automatically, but this means that changes to the cluster that require me to do things that cause decommissioning the node¹ generally then require an extra +30 minutes of waiting for nodes to be declared dead.
I'd like to just roll through the node decom/recreate process as fast as the hardware involved will allow.
**To Reproduce**
What did you do? Describe in your own words.
If possible, provide steps to reproduce the behavior:
1. Set up CockroachDB cluster
2. Drain a node.
3. Attempt to decommission it in the period of time between "suspect" and "dead"
4. Get the above error.
5. (be forced to wait for "dead")
**Expected behavior**
There should be a way to gracefully decommission nodes? There are lots of docs, but they focus on draining, but not on decommissioning.
For example, while [these docs](https://www.cockroachlabs.com/docs/stable/node-shutdown#decommissioning-and-draining-on-kubernetes) offer a high-level overview … they never get to the Cockroach specifics of decommissioning. (I can handle the k8s side; that doesn't have to be documented to perfection.)
But the "Node Shutdown" page … never runs `… node decommission …"!
**Additional context**
¹(in this case, resizing the disk, as cloud providers do not provide a general way to resize disks)
Jira issue: CRDB-36719
Contributor guide
Assessment
This issue has not been assessed yet.