zalando / zalando/postgres-operator
Issue with failover
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
I have installed the latest Zalando Postgresql operator (1.5.0) and tried to create the Postgresql DB cluster using the updated manifest.
I was trying to test out the failover with the latest docker image registry.opensource.zalan.do/acid/spilo-12:1.6-p3. It's unfortunate the failover is not happening as expected.
I have 2 replicas running for my DB and I have deleted the pod which was acting as a master, the failover didn't happen. Instead got the below error, but same is not happening with registry.opensource.zalan.do/acid/spilo-12:1.6-p2 docker image.
With an old docker image, when the master pod goes down, the standby pod takes the master role.
/var/run/postgresql:5432 - rejecting connections
2020-05-26 13:25:53,222 INFO: Lock owner: None; I am test-backup-test-test-postgresql-0
2020-05-26 13:25:53,222 INFO: Still starting up as a standby.
2020-05-26 13:25:53,223 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:25:53,776 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:25:53,821 WARNING: Retry got exception: 'connection problems'
2020-05-26 13:25:53,821 INFO: Error communicating with PostgreSQL. Will try again later
/var/run/postgresql:5432 - rejecting connections
2020-05-26 13:26:03,221 INFO: Lock owner: None; I am test-backup-test-test-postgresql-0
2020-05-26 13:26:03,222 INFO: Still starting up as a standby.
2020-05-26 13:26:03,223 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:26:04,016 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:26:04,017 WARNING: Retry got exception: 'connection problems'
2020-05-26 13:26:04,021 INFO: Error communicating with PostgreSQL. Will try again later
/var/run/postgresql:5432 - rejecting connections
2020-05-26 13:26:13,221 INFO: Lock owner: None; I am test-backup-test-test-postgresql-0
2020-05-26 13:26:13,221 INFO: Still starting up as a standby.
2020-05-26 13:26:13,222 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:26:13,424 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:26:13,426 WARNING: Retry got exception: 'connection problems'
2020-05-26 13:26:13,426 INFO: Error communicating with PostgreSQL. Will try again later
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failover with the updated manifest and the spilo-12:1.6-p3 image, then compare it with spilo-12:1.6-p2. Use the reported Patroni and PostgreSQL log messages to trace why the standby remains in startup; done means the standby assumes the master role after the master pod is deleted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes, postgresql
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100