cockroachdb / cockroachdb/cockroach-operator

Node stuck in `DECOMMISSIONING` state when upscale interrupts an ongoing downscale

Open
#1,138 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
318
Forks
104
Avg merge
1d 6h
Merged PRs (30d)
1

Description

### Description

A CockroachDB node can become permanently stuck in the `DECOMMISSIONING` membership state if the user scales up the cluster (`cockroach_cr.nodes`) while a previous downscale operation is still in progress.

The operator logic for `ReconcileDecommssion` relies on the condition `currentReplicas > cr.nodes`. If a user upscales (making `cr.nodes > currentReplicas`) while a node is in the intermediate `DECOMMISSIONING` state, the operator exits the reconcile loop for decommissioning. Consequently, the node is neither fully decommissioned nor returned to `ACTIVE` status.

### Root Cause Analysis

The issue lies in the entry conditions for the `ReconcileDecommssion` action. The operator triggers this action only if:

1. The Cluster is initialized.
2. `stsStatus.replicas == stsStatus.currentReplicas`
3. `stsStatus.currentReplicas > cockroach_cr.nodes` (Intent to downscale).

**The Failure Scenario:**

1. User initiates downscale (e.g., 5 -> 3). Condition (3) is met.
2. Operator calls `Decommission(node)`. The node enters the `DECOMMISSIONING` state.
3. `Decommission(node)` returns an error (e.g., network timeout, data moving too slowly). The operator returns early to retry later.
4. User initiates upscale (e.g., 3 -> 5).
5. On the next reconcile, Condition (3) evaluates to `false` because `currentReplicas` is no longer greater than `cr.replicas`.
6. The operator skips the `ReconcileDecommssion` block entirely. The node remains `DECOMMISSIONING` indefinitely.

### Steps to Reproduce

**Prerequisites:**

* A running CockroachDB cluster managed by the operator (5 nodes).
* (Optional) **ChaosMesh** installed to inject network faults.

1. **Initialize Workload:**
Run the `movr` workload to generate sufficient data:
```bash
cockroach workload init movr --num-histories 1000000 --num-rides 100000 --num-users 100000 --num-vehicles 100000
```

2. **Inject Fault:**
Use ChaosMesh to limit the Pod bandwidth to 1kbps. This ensures the decommissioning process stalls or errors out due to slow data replication.
3. **Trigger Downscale:**
Update the `CockroachDB` CR to reduce the replica count (e.g., 5 -> 3).
4. **Verify State:**
Wait until the target node enters the `DECOMMISSIONING` state:
```bash
cockroach node status --insecure --decommission

```

5. **Trigger Upscale:**
Immediately update the `CockroachDB` CR to increase the replica count (e.g., back to 5 or higher).

### **Observed Behavior**

The node previously targeted for removal remains in the `DECOMMISSIONING` state while new nodes are added. It does not revert to `ACTIVE`.

**Log Output:**

```text
bash-5.1$ cockroach node status --insecure --decommission
id | address | sql_address | build | started_at | updated_at | locality | attrs | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------------------------------------------------+-----------------------------------------------------------+---------+--------------------------------------+--------------------------------------+----------+-------+--------------+---------+-------------------+--------------------+-----------------+--------------
1 | cockroachdb-0.cockroachdb.cockroach-operator-system:26258 | cockroachdb-0.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:54:10.174444 +0000 UTC | 2026-01-12 03:07:17.954084 +0000 UTC | | [] | true | true | 104 | false | active | false
2 | cockroachdb-1.cockroachdb.cockroach-operator-system:26258 | cockroachdb-1.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:55:32.643908 +0000 UTC | 2026-01-12 03:07:17.797383 +0000 UTC | | [] | true | true | 98 | false | active | false
3 | cockroachdb-2.cockroachdb.cockroach-operator-system:26258 | cockroachdb-2.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:55:34.215002 +0000 UTC | 2026-01-12 03:07:16.41831 +0000 UTC | | [] | true | true | 100 | false | active | false
4 | cockroachdb-4.cockroachdb.cockroach-operator-system:26258 | cockroachdb-4.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:39:19.128573 +0000 UTC | 2026-01-12 03:07:16.193822 +0000 UTC | | [] | true | true | 40 | true | decommissioning | false
5 | cockroachdb-3.cockroachdb.cockroach-operator-system:26258 | cockroachdb-3.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:56:23.412081 +0000 UTC | 2026-01-12 03:07:18.188583 +0000 UTC | | [] | true | true | 104 | false | active | false
(5 rows)
```

### Expected Behavior

If the operator detects that the desired replica count has increased (upscale) while a node is currently `DECOMMISSIONING`:

1. The operator should detect the intermediate state.
2. It should explicitly recommission the node (cancel the decommission) to return it to `ACTIVE` status before proceeding with the upscale.

### Severity
Major
(But this is not a production failure)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the ReconcileDecommisssion action and its entry conditions, then trace the Decommission call and the node membership state handling. Reproduce the downscale interruption followed by an upscale if the environment permits, and inspect existing reconciliation tests. Done means an interrupted decommissioning node is handled during an upscale and returns to ACTIVE without preventing the scale-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.