cloudnative-pg / cloudnative-pg/cloudnative-pg

[Bug]: Node cordon (unschedulable taint) causes a switchover

Open
#7,799 20 comments 7 reactions 1 assignee Claimed by @gbartolini View on GitHub
no-stale
Dominant language
Go
Stars
9.3k
Forks
759
Avg merge
2d 6h
Merged PRs (30d)
44

Description

### Is there an existing issue already for this bug?

- [x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

### I have read the troubleshooting guide

- [x] I have read the troubleshooting guide and I think this is a new bug.

### I am running a supported version of CloudNativePG

- [x] I have read the troubleshooting guide and I think this is a new bug.

### Contact Details

_No response_

### Version

1.26 (latest patch)

### What version of Kubernetes are you using?

1.31

### What is your Kubernetes environment?

Other

### How did you install the operator?

Helm

### What happened?

Cordoning a kubernetes node causes the operator to perform a switchover, even if the primaryUpdateStrategy is set to "supervised". While I believe this is intended, I would still consider this "feature" as a bug, since you do not normally expect in a kubernetes environment, that a cordon affects existing workloads.

I'm aware, that you can turn on the maintenance window for a given database, but we manage a lot of database clusters, and manually turning these on every time we conduct some form of maintenance that requires a node cordon is suboptimal.

I'm also aware, that instead of cordoning (which is equivalent to adding the node.kubernetes.io/unschedulable:NoSchedule taint), you could add a custom taint with the NoSchedule effect, so you could achieve the same thing, while it being ignored by the operator. While this is a good workaround, I do not believe this should be the way to go, since it's not a thing people are generally aware of. Therefore, leaving taints on the nodes unintentionally could be common, or even worse, cordoning a node running a primary instance, and performing an unintended switchover.

While a switchover normally causes little to negligible downtime, in some cases, this could very much disturb sensitive workloads, so I believe at least there should be an option (ideally, on the operator level) to disable automated switchover, if a cordon is issued on a node.

### Cluster resource

```shell

```

### Relevant log output

```shell
{"level":"info","ts":"2025-06-10T07:19:30.066361654Z","msg":"Primary is running on an unschedulable node, will try switching over","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKi
nd":"Cluster","Cluster":{"name":"cluster","namespace":"cluster-namespace"},"namespace":"cluster-namespace","name":"cluster","reconcileID":"9b6e086d-48a6-4e62-8799-9a2634a2c419","node":"node1","primary":"cluster-1"}
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.