cockroachdb / cockroachdb/cockroach

Decommission fails due to fixed-column CSV parsing in cockroach-operator with newer CockroachDB output

Open
#163,578 2 comments 0 reactions 0 assignees View on GitHub
C-bug O-community T-cloud-platform
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Problem Description**

When using CockroachDB `v25.4.3`, scale-up works (`3 -> 5`) but scale-down (`5 -> 3`) gets stuck during decommission.

After scaling down, pod `cockroachdb-4` stays at `0/1 Running` indefinitely, and the operator repeatedly logs errors such as:

- `unexpected node status`
- `could not find the id of replica 4`

Observed pod state:

```bash
kubectl get pods
NAME READY STATUS RESTARTS AGE
cockroachdb-0 1/1 Running 0 3m1s
cockroachdb-1 1/1 Running 0 3m1s
cockroachdb-2 1/1 Running 0 3m1s
cockroachdb-3 1/1 Running 0 2m15s
cockroachdb-4 0/1 Running 0 2m15s
```

Operator log includes:

```text
could not find the id of replica 4
```

---

**To Reproduce**

1. Deploy cockroach-operator on Kubernetes.
2. Create a CRDB cluster with `spec.nodes=3` and image `cockroachdb/cockroach:v25.4.3`.
3. Scale up to 5:
```bash
kubectl -n default patch crdbcluster cockroachdb --type=merge -p '{"spec":{"nodes":5}}'
```
4. Wait until new pods appear/running.
5. Scale down to 3:
```bash
kubectl -n default patch crdbcluster cockroachdb --type=merge -p '{"spec":{"nodes":3}}'
```
6. Observe operator logs and pod status:
- pod `cockroachdb-4` remains `0/1 Running`
- operator logs decommission-related errors.

---

**Expected behavior**

Scale-down should complete successfully:
- decommission extra nodes,
- remove extra replicas/pods,
- converge to a healthy 3-node cluster.

---

**Additional data / screenshots**

In `cockroach-operator/pkg/scale/drainer.go`, decommission status parsing uses fixed CSV column indexes from:

```text
cockroach node status --decommission --format=csv
```

Current code assumes:
- `record[8]` => `is_live`
- `record[9]` => `replicas`
- `record[10]` => `is_decommissioning`

If CockroachDB CSV schema shifts (column insertion/reordering), these indexes no longer map to the intended fields, the operator will give the followingi error:
- `unexpected node status` (`pkg/scale/drainer.go`)
- and later `could not find the id of replica 4` (`pkg/scale/drainer.go`).

A header-name-based parse (instead of fixed indices) would likely be more robust.

---

**Environment:**
- CockroachDB version: `v25.4.3` (fails), `v25.2.12` (works in my test)
- CockroachDB operator version: built from main at commit `b4cfbb4` (commit date: January 29, 2026)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.