zalando / zalando/postgres-operator
Single Standby moved away from "unschedulable" Node (only elegible node)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
- Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.9.0
- Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
- Are you running Postgres Operator in production? yes
- Type of issue? Bug report
I'm running multiple standby only clusters with a single node. Once the host where the node is located is cordoned, it will try to move away the pods, and will restart them on the same node again.
Procedure:
Cordon host
Operator deletes pod
Sts recreates pod on the same node
Since the pod cannot be started on another node due to storage constraints (local disk), it should not be moved away.
Since the pod is a replica and not a master, I think that the master logic should not apply.
Logs from operator:
time="2023-03-30T07:59:41Z" level=info msg="moving pods: node \"/at-host-40\" became unschedulable and does not have a ready label: map[]" pkg=controller
time="2023-03-30T07:59:41Z" level=info msg="starting process to migrate master pod \"cf-database/platon-standby01-0\"" cluster-name=cf-database/platon-standby01 pkg=cluster
time="2023-03-30T07:59:41Z" level=warning msg="migrating single pod cluster \"cf-database/platon-standby01\", this will cause downtime of the Postgres cluster until pod is back" cluster-name=cf-database/platon-standby01 pkg=cluster
time="2023-03-30T07:59:41Z" level=info msg="moving pod \"cf-database/platon-standby01-0\" out of the end-of-life node \"at-host-40\"" cluster-name=cf-database/platon-standby01 pkg=cluster
I see there's an option with ready labels to work around this check, how would it work if the node became NotReady (i.e. down)?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the operator's pkg=controller and pkg=cluster paths shown in the logs, then reproduce the cordon sequence for a single-standby cluster using local storage. Trace why the standby is treated as a master and moved from the only eligible node. Done means the sole standby is not unnecessarily moved when no alternative node can satisfy its storage constraints, with the NotReady case addressed or explicitly covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, postgresql
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100