Scale-down reconciliation gets stuck after CockroachDB node has been decommissioned

# Scale-down reconciliation gets stuck after CockroachDB node has been decommissioned

## Summary

When scaling down a `CrdbCluster`, the operator may enter an infinite reconciliation loop after the target CockroachDB node has already been decommissioned.

In our case:

- Scaling from **5 → 4** starts normally.
- The target CockroachDB node is successfully decommissioned.
- The operator then repeatedly reports:

```
could not find the id of replica 4
```

and retries forever.

---

## Environment

- CockroachDB Operator: **v2.18.3**
- CockroachDB: **v25.4.2**
- Kubernetes: **v1.33.3**

### CrdbCluster configuration

```yaml
spec:
nodes: 4

image:
name: image.ccos.io/cockroachdb/cockroach:v25.4.2
pullPolicy: IfNotPresent

tlsEnabled: true

grpcPort: 26258
sqlPort: 26257
httpPort: 8080

maxUnavailable: 1

resources:
requests:
cpu: 500m
memory: 2Gi
limits:
cpu: "2"
memory: 8Gi

dataStore:
supportsAutoResize: false
pvc:
spec:
accessModes:
- ReadWriteOnce
storageClassName: ccos-hostpath-data-stor
volumeMode: Filesystem
resources:
requests:
storage: 60Gi
```

---

## Steps to reproduce

1. Start with a healthy 5-node cluster.
2. Update `CrdbCluster.spec.nodes` from **5** to **4**.
3. Wait for the operator to perform the scale-down.

---

## Observed behavior

The operator starts decommissioning `cockroachdb-4`.

The Pod eventually becomes:

```
0/1 Running
```

Meanwhile:

- `cockroach node status` changes from

```
2 3 4 7 9
```

to

```
2 3 4 7
```

indicating that the target CockroachDB node (NodeID **9**) has already disappeared.

`crdb_internal.gossip_nodes` still contains the node but reports:

- `is_live = false`
- `ranges = 0`
- `leases = 0`

The operator then repeatedly logs:

```
could not find the id of replica 4
```

from `pkg/actor/decommission.go`.

The reconciliation loop never converges and continues retrying indefinitely.

---

## Expected behavior

The operator should complete the scale-down once the target node has been successfully decommissioned.

If the node can no longer be resolved, it should either:

- treat the decommission as already completed, or
- return a terminal, actionable error instead of retrying forever.

---

## Additional context

This cluster previously experienced another scale-down failure (5 → 3) with a different error (`unexpected node status`).

That issue required manually deleting the Pod (and its PVC) before reconciliation could continue. After recovery, the recreated Pod joined as a new CockroachDB node, resulting in non-contiguous node IDs:

| Pod | Node ID |
|------|---------|
| cockroachdb-0 | 7 |
| cockroachdb-1 | 2 |
| cockroachdb-2 | 3 |
| cockroachdb-3 | 4 |
| cockroachdb-4 | 9 |

We do not know whether this earlier incident is related, but include it in case it is helpful for debugging.

---

## Attachments

- Operator log covering the scale-down and repeated reconciliation failures.

[operator.log](https://github.com/user-attachments/files/30290726/operator.log)

Start in pkg/actor/decommission.go, where the repeated “could not find the id of replica 4” error is logged, and compare the decommission flow with the reported cockroach node status and gossip_nodes state. Reproduce a 5-to-4 scale-down if possible using the attached operator log. Done means reconciliation converges after the target node disappears, either treating decommission as complete or returning a terminal actionable error.

Written by the indexing model from the issue text.