cockroachdb / cockroachdb/cockroach
Decommission fails due to fixed-column CSV parsing in cockroach-operator with newer CockroachDB output
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Problem Description**
When using CockroachDB `v25.4.3`, scale-up works (`3 -> 5`) but scale-down (`5 -> 3`) gets stuck during decommission.
After scaling down, pod `cockroachdb-4` stays at `0/1 Running` indefinitely, and the operator repeatedly logs errors such as:
- `unexpected node status`
- `could not find the id of replica 4`
Observed pod state:
```bash
kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroachdb-0 1/1 Running 0 3m1s
cockroachdb-1 1/1 Running 0 3m1s
cockroachdb-2 1/1 Running 0 3m1s
cockroachdb-3 1/1 Running 0 2m15s
cockroachdb-4 0/1 Running 0 2m15s
```
Operator log includes:
```text
could not find the id of replica 4
```
---
**To Reproduce**
1. Deploy cockroach-operator on Kubernetes.
2. Create a CRDB cluster with `spec.nodes=3` and image `cockroachdb/cockroach:v25.4.3`.
3. Scale up to 5:
```bash
kubectl -n default patch crdbcluster cockroachdb --type=merge -p '{"spec":{"nodes":5}}'
```
4. Wait until new pods appear/running.
5. Scale down to 3:
```bash
kubectl -n default patch crdbcluster cockroachdb --type=merge -p '{"spec":{"nodes":3}}'
```
6. Observe operator logs and pod status:
- pod `cockroachdb-4` remains `0/1 Running`
- operator logs decommission-related errors.
---
**Expected behavior**
Scale-down should complete successfully:
- decommission extra nodes,
- remove extra replicas/pods,
- converge to a healthy 3-node cluster.
---
**Additional data / screenshots**
In `cockroach-operator/pkg/scale/drainer.go`, decommission status parsing uses fixed CSV column indexes from:
```text
cockroach node status --decommission --format=csv
```
Current code assumes:
- `record[8]` => `is_live`
- `record[9]` => `replicas`
- `record[10]` => `is_decommissioning`
If CockroachDB CSV schema shifts (column insertion/reordering), these indexes no longer map to the intended fields, the operator will give the followingi error:
- `unexpected node status` (`pkg/scale/drainer.go`)
- and later `could not find the id of replica 4` (`pkg/scale/drainer.go`).
A header-name-based parse (instead of fixed indices) would likely be more robust.
---
**Environment:**
- CockroachDB version: `v25.4.3` (fails), `v25.2.12` (works in my test)
- CockroachDB operator version: built from main at commit `b4cfbb4` (commit date: January 29, 2026)
Contributor guide
Assessment
This issue has not been assessed yet.