zalando / zalando/postgres-operator

Issue with failover

Open
#993 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

spilo
Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

I have installed the latest Zalando Postgresql operator (1.5.0) and tried to create the Postgresql DB cluster using the updated manifest.

I was trying to test out the failover with the latest docker image registry.opensource.zalan.do/acid/spilo-12:1.6-p3. It's unfortunate the failover is not happening as expected.

I have 2 replicas running for my DB and I have deleted the pod which was acting as a master, the failover didn't happen. Instead got the below error, but same is not happening with registry.opensource.zalan.do/acid/spilo-12:1.6-p2 docker image.

With an old docker image, when the master pod goes down, the standby pod takes the master role.

/var/run/postgresql:5432 - rejecting connections
2020-05-26 13:25:53,222 INFO: Lock owner: None; I am test-backup-test-test-postgresql-0
2020-05-26 13:25:53,222 INFO: Still starting up as a standby.
2020-05-26 13:25:53,223 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:25:53,776 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:25:53,821 WARNING: Retry got exception: 'connection problems'
2020-05-26 13:25:53,821 INFO: Error communicating with PostgreSQL. Will try again later
/var/run/postgresql:5432 - rejecting connections
2020-05-26 13:26:03,221 INFO: Lock owner: None; I am test-backup-test-test-postgresql-0
2020-05-26 13:26:03,222 INFO: Still starting up as a standby.
2020-05-26 13:26:03,223 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:26:04,016 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:26:04,017 WARNING: Retry got exception: 'connection problems'
2020-05-26 13:26:04,021 INFO: Error communicating with PostgreSQL. Will try again later
/var/run/postgresql:5432 - rejecting connections
2020-05-26 13:26:13,221 INFO: Lock owner: None; I am test-backup-test-test-postgresql-0
2020-05-26 13:26:13,221 INFO: Still starting up as a standby.
2020-05-26 13:26:13,222 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:26:13,424 INFO: establishing a new patroni connection to the postgres cluster
2020-05-26 13:26:13,426 WARNING: Retry got exception: 'connection problems'
2020-05-26 13:26:13,426 INFO: Error communicating with PostgreSQL. Will try again later

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failover with the updated manifest and the spilo-12:1.6-p3 image, then compare it with spilo-12:1.6-p2. Use the reported Patroni and PostgreSQL log messages to trace why the standby remains in startup; done means the standby assumes the master role after the master pod is deleted.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.