cockroachdb / cockroachdb/cockroach-operator
Node stuck in `DECOMMISSIONING` state when upscale interrupts an ongoing downscale
- Dominant language
- Go
- Stars
- 318
- Forks
- 104
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
### Description
A CockroachDB node can become permanently stuck in the `DECOMMISSIONING` membership state if the user scales up the cluster (`cockroach_cr.nodes`) while a previous downscale operation is still in progress.
The operator logic for `ReconcileDecommssion` relies on the condition `currentReplicas > cr.nodes`. If a user upscales (making `cr.nodes > currentReplicas`) while a node is in the intermediate `DECOMMISSIONING` state, the operator exits the reconcile loop for decommissioning. Consequently, the node is neither fully decommissioned nor returned to `ACTIVE` status.
### Root Cause Analysis
The issue lies in the entry conditions for the `ReconcileDecommssion` action. The operator triggers this action only if:
1. The Cluster is initialized.
2. `stsStatus.replicas == stsStatus.currentReplicas`
3. `stsStatus.currentReplicas > cockroach_cr.nodes` (Intent to downscale).
**The Failure Scenario:**
1. User initiates downscale (e.g., 5 -> 3). Condition (3) is met.
2. Operator calls `Decommission(node)`. The node enters the `DECOMMISSIONING` state.
3. `Decommission(node)` returns an error (e.g., network timeout, data moving too slowly). The operator returns early to retry later.
4. User initiates upscale (e.g., 3 -> 5).
5. On the next reconcile, Condition (3) evaluates to `false` because `currentReplicas` is no longer greater than `cr.replicas`.
6. The operator skips the `ReconcileDecommssion` block entirely. The node remains `DECOMMISSIONING` indefinitely.
### Steps to Reproduce
**Prerequisites:**
* A running CockroachDB cluster managed by the operator (5 nodes).
* (Optional) **ChaosMesh** installed to inject network faults.
1. **Initialize Workload:**
Run the `movr` workload to generate sufficient data:
```bash
cockroach workload init movr --num-histories 1000000 --num-rides 100000 --num-users 100000 --num-vehicles 100000
```
2. **Inject Fault:**
Use ChaosMesh to limit the Pod bandwidth to 1kbps. This ensures the decommissioning process stalls or errors out due to slow data replication.
3. **Trigger Downscale:**
Update the `CockroachDB` CR to reduce the replica count (e.g., 5 -> 3).
4. **Verify State:**
Wait until the target node enters the `DECOMMISSIONING` state:
```bash
cockroach node status --insecure --decommission
```
5. **Trigger Upscale:**
Immediately update the `CockroachDB` CR to increase the replica count (e.g., back to 5 or higher).
### **Observed Behavior**
The node previously targeted for removal remains in the `DECOMMISSIONING` state while new nodes are added. It does not revert to `ACTIVE`.
**Log Output:**
```text
bash-5.1$ cockroach node status --insecure --decommission
id | address | sql_address | build | started_at | updated_at | locality | attrs | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------------------------------------------------+-----------------------------------------------------------+---------+--------------------------------------+--------------------------------------+----------+-------+--------------+---------+-------------------+--------------------+-----------------+--------------
1 | cockroachdb-0.cockroachdb.cockroach-operator-system:26258 | cockroachdb-0.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:54:10.174444 +0000 UTC | 2026-01-12 03:07:17.954084 +0000 UTC | | [] | true | true | 104 | false | active | false
2 | cockroachdb-1.cockroachdb.cockroach-operator-system:26258 | cockroachdb-1.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:55:32.643908 +0000 UTC | 2026-01-12 03:07:17.797383 +0000 UTC | | [] | true | true | 98 | false | active | false
3 | cockroachdb-2.cockroachdb.cockroach-operator-system:26258 | cockroachdb-2.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:55:34.215002 +0000 UTC | 2026-01-12 03:07:16.41831 +0000 UTC | | [] | true | true | 100 | false | active | false
4 | cockroachdb-4.cockroachdb.cockroach-operator-system:26258 | cockroachdb-4.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:39:19.128573 +0000 UTC | 2026-01-12 03:07:16.193822 +0000 UTC | | [] | true | true | 40 | true | decommissioning | false
5 | cockroachdb-3.cockroachdb.cockroach-operator-system:26258 | cockroachdb-3.cockroachdb.cockroach-operator-system:26257 | v25.4.2 | 2026-01-12 02:56:23.412081 +0000 UTC | 2026-01-12 03:07:18.188583 +0000 UTC | | [] | true | true | 104 | false | active | false
(5 rows)
```
### Expected Behavior
If the operator detects that the desired replica count has increased (upscale) while a node is currently `DECOMMISSIONING`:
1. The operator should detect the intermediate state.
2. It should explicitly recommission the node (cancel the decommission) to return it to `ACTIVE` status before proceeding with the upscale.
### Severity
Major
(But this is not a production failure)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the ReconcileDecommisssion action and its entry conditions, then trace the Decommission call and the node membership state handling. Reproduce the downscale interruption followed by an upscale if the environment permits, and inspect existing reconciliation tests. Done means an interrupted decommissioning node is handled during an upscale and returns to ACTIVE without preventing the scale-up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100