zalando / zalando/postgres-operator
moveMasterPodsOffNode gives up after one failed retry instead of retrying
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
Summary
When a node hosting a Postgres master pod is cordoned, the operator's moveMasterPodsOffNode is supposed to retry finding a Patroni switchover candidate for up to master_pod_move_timeout (default 20m, checked every 1m). In practice it gets exactly one attempt (~4-5s) and then permanently gives up for that node, because retryutil.RetryWorker treats any non-nil error from the retried function as terminal rather than retryable.
If no synced standby exists in that first ~4-second window (e.g. because a replica pod was recreated moments earlier by an unrelated rolling update), the master is never migrated off the node. The node drain then falls back to the plain Kubernetes eviction API, which the Postgres master PodDisruptionBudget (minAvailable: 1, selector spilo-role=master) correctly and indefinitely rejects — since nothing ever moves the master label to another pod. The drain then blocks until an external timeout (e.g. the node-drain timeout used by whatever machine-management layer is orchestrating the drain) forces the node replacement through.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating moveMasterPodsOffNode and retryutil.RetryWorker, then trace how master_pod_move_timeout and its one-minute interval are passed into the retry logic. Reproduce the no-synced-standby case and verify that the operator keeps looking for a Patroni switchover candidate until the timeout instead of stopping after the first error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, postgresql
- Domain
- databases, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100