[Bug]: Replicas PDB becomes ineffective during switchover on node drain - NoPods event allows eviction of degraded cluster

### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

tob@symprex.com

### Version

1.28 (latest patch)

### What version of Kubernetes are you using?

1.34

### What is your Kubernetes environment?

Cloud: Azure AKS

### How did you install the operator?

Helm

### What happened?

### Summary

During an AKS Kubernetes node upgrade, the replicas PodDisruptionBudget fires a `NoPods` event and ceases to block evictions, resulting in a rougly 1 minute window where the cluster runs with only the primary instance and zero replicas. This occurs because the PDB's label selector (`cnpg.io/instanceRole: replica`) temporarily matches zero running pods during a switchover triggered by the node cordon.

### Expected behaviour

The PDB should ensure that at least 1 replica remains available at all times during a node drain, preventing the cluster from being reduced to a single instance.

### Actual behaviour

The replicas PDB reports `NoPods` ("No matching pods found") during a switchover, allowing AKS to drain the demoted primary immediately - before any replica has recovered. The cluster runs with a single instance (the new primary) for ~1 min.

### Environment

- 3-instance CNPG cluster with `enablePDB: true`, `primaryUpdateMethod: switchover`
- `minSyncReplicas: 2`, `maxSyncReplicas: 2`
- `podAntiAffinity: required` (1 pod per node) + `topologySpreadConstraints` (1 per zone)
- 3 AKS nodes across 3 availability zones (1 node per AZ)
- PVCs using `PremiumV2_LRS` (Azure Premium SSD v2, LRS only - zone-pinned at provisioning time via `WaitForFirstConsumer`)
- AKS node pool upgrade settings: `maxSurge: 0`, `maxUnavailable: 1`, `nodeSoakDuration: 5 min`

## Root cause analysis

### DISCLAIMER: I'm fully aware this looks AI generated, because it is. However, it is an accurate description of the probelm we've experienced. I used AI to analyse log files from cnpg and kubernetes. I included it as it might be useful to fix the issue described above.

The problem is a **label-transition race condition** involving three CNPG functions:

#### 1. Proactive switchover on node cordon

When AKS cordons a node during an upgrade, it sets `spec.unschedulable: true`, which adds the `node.kubernetes.io/unschedulable` taint. This taint is included in CNPG's default `DRAIN_TAINTS` list. The operator detects this via `isNodeUnschedulableOrBeingDrained()` in `internal/controller/replicas.go` and immediately calls `setPrimaryOnSchedulableNode()` to move the primary to a schedulable node.

#### 2. `updateRoleLabels()` skips inactive pods

After the switchover, `ReconcileMetadata()` calls `updateRoleLabels()` (`pkg/reconciler/instance/metadata.go`) to update the `cnpg.io/instanceRole` labels on all pods. However, this function has a guard:

```go
if !utils.IsPodActive(*instance) {
contextLogger.Trace("Ignoring not active Pod during label update",
"pod", instance.Name, "status", instance.Status)
return false
}
```

`IsPodActive()` (`pkg/utils/pod_conditions.go`) returns `false` for pods in `Pending` phase (e.g., `FailedScheduling`) or pods with a `DeletionTimestamp` (being terminated):

```go
func IsPodActive(p corev1.Pod) bool {
return corev1.PodSucceeded != p.Status.Phase &&
corev1.PodPending != p.Status.Phase &&
corev1.PodFailed != p.Status.Phase &&
p.DeletionTimestamp == nil
}
```

#### 3. PDB with zero matching pods does not block evictions

The replicas PDB is defined as:

```yaml
selector:
matchLabels:
cnpg.io/cluster: test-pg-cluster
cnpg.io/instanceRole: replica
minAvailable: 1
```

When zero running pods carry the `cnpg.io/instanceRole: replica` label, the PDB's `expectedPods` becomes 0. A PDB with `minAvailable: 1` and 0 expected pods is vacuously satisfied - Kubernetes does not block any evictions.

### The specific failure sequence

Given 3 instances: `1` (primary on node-3), `2` (replica on node-1), `3` (replica on node-2):

| Time | Event | Pod States | Replicas PDB |
|------|-------|-----------|--------------|
| T+0 | AKS cordons node-3 | `1`: primary ✓, `2`: replica ✓, `3`: Pending (FailedScheduling - zone-2 node still being reimaged from previous cycle) | Matches `2` only (1 replica). `3` skipped by `updateRoleLabels()` (Pending). |
| T+7s | CNPG switchover: `1` → `2` | `1`: restarting (role change), `2`: primary ✓, `3`: Pending | `2` label changed to `primary`. `1` restarting (not active). `3` Pending (not active). **Matched: 0 replicas** |
| T+10s | AKS drains `1` from node-3 | `1`: killed, `2`: primary ✓, `3`: Pending | **NoPods event**. Drain proceeds unblocked. |
| T+72s | `3` scheduled on reimaged node-2 | `1`: Pending (waiting for node-3), `2`: primary ✓, `3`: replica ✓ | `3` matched. Back to 1 replica. |

**Result:** 62-second window with only the primary running and zero replicas. With `minSyncReplicas: 2`, PostgreSQL cannot satisfy the synchronous replication requirement during this window - writes stall.

### Observed events

```
2026-03-02T12:30:07Z Cluster/test-pg-cluster SwitchingOver Current primary is running on unschedulable node ..., switching over from test-pg-cluster-1 to test-pg-cluster-2
2026-03-02T12:30:11Z Pod/test-pg-cluster-1 Unhealthy Startup probe failed: HTTP probe failed with statuscode: 500
2026-03-02T12:30:17Z Pod/test-pg-cluster-1 Killing Stopping container postgres
2026-03-02T12:30:18Z PodDisruptionBudget/test-pg-cluster NoPods No matching pods found
```

### Impact amplification with LRS PVCs

This issue is particularly severe with Azure `PremiumV2_LRS` (and other LRS) storage because:
- PVCs are zone-pinned at provisioning time via `WaitForFirstConsumer`
- Premium SSD v2 **does not support ZRS** - only LRS is available
- When a node is drained, the evicted pod can only schedule in the same AZ
- With 1 node per AZ and `maxSurge: 0`, the pod must wait for its specific node to be reimaged - leading to extended `FailedScheduling` periods (5-8 minutes)
- During this entire `FailedScheduling` period, `IsPodActive()` returns false, `updateRoleLabels()` skips the pod, and the PDB may have reduced or zero matching pods

### Suggested fix approaches

#### Approach 1: Add a cluster-level PDB

In addition to (or replacing) the two role-specific PDBs, create a third PDB that uses the cluster label without the role filter:

```go
// In pkg/specs/poddisruptionbudget.go

func BuildClusterPodDisruptionBudget(cluster *apiv1.Cluster) *policyv1.PodDisruptionBudget {
if cluster == nil || cluster.Spec.Instances < 3 {
return nil
}
minAvailable := intstr.FromInt32(2) // Always keep at least 2 instances alive

pdb := &policyv1.PodDisruptionBudget{
ObjectMeta: metav1.ObjectMeta{
Name: cluster.Name + "-cluster",
Namespace: cluster.Namespace,
},
Spec: policyv1.PodDisruptionBudgetSpec{
Selector: &metav1.LabelSelector{
MatchLabels: map[string]string{
utils.ClusterLabelName: cluster.Name,
// No instanceRole filter - matches all instances
},
},
MinAvailable: &minAvailable,
},
}
cluster.SetInheritedDataAndOwnership(&pdb.ObjectMeta)
return pdb
}
```

**Pros:** Immune to label transitions - the `cnpg.io/cluster` label never changes. Always protects the overall instance count regardless of role.

**Cons:** May need to coordinate with the existing primary and replica PDBs so that the combined PDB set doesn't overly restrict evictions. During a maintenance window with `reusePVC: true`, this PDB would also need to be relaxed (similar to how the replica PDB is currently deleted).

**Note:** The `minAvailable` value would need to be calculated based on the cluster's instance count and possibly the number of `minSyncReplicas` to ensure writes don't stall. For a 3-instance cluster, `minAvailable: 2` ensures at least 1 primary + 1 replica are always running.

#### Approach 2: Pre-check replica availability before switchover

In `reconcileTargetPrimaryFromPods()` (`internal/controller/replicas.go`), before calling `setPrimaryOnSchedulableNode()`, verify that at least one other instance is Running and Active to serve as a replica after the switchover:

```go
if isPrimaryOnUnschedulableNode {
// Count active, running replicas that are NOT on unschedulable nodes
activeReplicas := 0
for _, item := range status.Items {
if item.Pod.Name == primary.Pod.Name {
continue
}
if !utils.IsPodActive(item.Pod) {
continue
}
isOnUnschedulable, _ := r.isNodeUnschedulableOrBeingDrained(ctx, item.Node)
if !isOnUnschedulable {
activeReplicas++
}
}

if activeReplicas == 0 {
contextLogger.Info("Deferring switchover: no active replicas on schedulable nodes",
"primary", primary.Pod.Name, "node", primary.Node)
// Requeue - check again later when a replica becomes available
return "", nil
}

return r.setPrimaryOnSchedulableNode(ctx, cluster, status, &primary)
}
```

**Pros:** Directly addresses the root cause by delaying the switchover until at least one replica is available. This prevents the label transition that causes the NoPods event. Does not require additional PDB resources.

**Cons:** If the primary's node is truly being drained (not just cordoned), delaying the switchover means the primary PDB will block the drain. This is arguably the correct behaviour - the PDB should block the drain until the cluster is ready to handle the switchover safely. However, this could cause the drain to timeout if the replica takes too long to recover (e.g., extended `FailedScheduling` due to LRS zone-pinning).

A timeout or maximum-deferral-count could be added to avoid indefinite blocking:

```go
const maxSwitchoverDeferrals = 10

deferralKey := fmt.Sprintf("switchover-deferrals-%s", primary.Pod.Name)
deferrals := cluster.Annotations[deferralKey]
deferralCount, _ := strconv.Atoi(deferrals)

if activeReplicas == 0 && deferralCount < maxSwitchoverDeferrals {
// Increment deferral count and requeue
// ...
return "", nil
}
// Proceed with switchover (either replicas available or max deferrals reached)
```

### Related issues

- #7799 - Node cordon (unschedulable taint) causes a switchover: discusses the broader issue of cordoning triggering switchovers
- #2570 - Allow configurable PDBs: led to the `enablePDB` feature but didn't address the label-gap issue

### Cluster resource

```shell
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: test-pg-cluster
namespace: postgres
spec:
instances: 3
primaryUpdateMethod: switchover
enablePDB: true
minSyncReplicas: 2
maxSyncReplicas: 2
failoverDelay: 0
affinity:
podAntiAffinityType: required
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
cnpg.io/cluster: test-pg-cluster
storage:
size: 10Gi
storageClass: managed-premiumv2-lrs
pvcTemplate:
accessModes:
- ReadWriteOnce
storageClassName: managed-premiumv2-lrs
```

### Relevant log output

```shell
# Switchover triggered by node cordon
{"level":"info","ts":"2026-03-02T12:30:07Z","msg":"Primary is running on an unschedulable node, will try switching over","node":"aks-pg-14405023-vmss000002","primary":"test-pg-cluster-1"}

# Cluster events during the 62-second window
2026-03-02T12:30:07Z Cluster/test-pg-cluster SwitchingOver Current primary is running on unschedulable node aks-pg-14405023-vmss000002, switching over from test-pg-cluster-1 to test-pg-cluster-2
2026-03-02T12:30:08Z Pod/test-pg-cluster-1 Created Created container: postgres (role change restart)
2026-03-02T12:30:11Z Pod/test-pg-cluster-1 Unhealthy Startup probe failed: HTTP probe failed with statuscode: 500
2026-03-02T12:30:17Z Pod/test-pg-cluster-1 Killing Stopping container postgres
2026-03-02T12:30:18Z PodDisruptionBudget/test-pg-cluster NoPods No matching pods found
2026-03-02T12:31:11Z Pod/test-pg-cluster-1 FailedScheduling 0/4 nodes are available: 1 node(s) were unschedulable, ...
2026-03-02T12:31:19Z Pod/test-pg-cluster-3 Ready Replica recovered on reimaged node

# Operator DRAIN_TAINTS config (default)
drainTaints: ["node.kubernetes.io/unschedulable","ToBeDeletedByClusterAutoscaler","karpenter.sh/disrupted","karpenter.sh/disruption"]
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct