zalando / zalando/postgres-operator

Postgres pod starts as secondary node instead of primary after Kubernetes cluster upgrade

Open
#2,288 9 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

  • Which version of the Postgres Operator are you using?
    registry.opensource.zalan.do/acid/postgres-operator:v1.9.0.

  • Where is the Postgres Operator running? Is it on cloud or bare metal? Are you using Kubernetes or OpenShift?
    The Postgres Operator is running on bare metal.

  • Are you currently using the Postgres Operator in production?
    Yes, the Postgres Operator is currently being used in production.

  • What type of issue are you facing?
    This is a bug report.

Description:
We upgraded our Kubernetes cluster from version 1.24.8 to 1.26.3 by following the official docs. When we drained the node running the Postgres pod, it would respawn on another node without any issues. However, the Postgres pod started as a secondary node instead of the primary node, even though we had configured it to run in 1-master mode. As a result, the database rejected all connections from the application. Please see the error log below for more information:

2023-04-14 10:26:25,574 INFO: Lock owner: None; I am mattermost-pgsql-0                                                                                                                    │
│ 2023-04-14 10:26:25,574 INFO: starting as a secondary                                                                                                                                      │
│ 2023-04-14 10:26:25,664 INFO: postmaster pid=85678 
│ 2023-04-12 10:26:25 UTC [15597]: [8-1] 6436b473.3ced 0     HINT:  Future log output will appear in directory "../pg_log".                                                                  │
│ /var/run/postgresql:5432 - rejecting connections                                                                                                                                           │
│ /var/run/postgresql:5432 - rejecting connections                                                                                                                                           │
│ /var/run/postgresql:5432 - rejecting connections                                                                                                                                           │
│ /var/run/postgresql:5432 - rejecting connections                                                                                                                                           │
│ /var/run/postgresql:5432 - no response 

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by reproducing the node-drain and pod rescheduling after the Kubernetes 1.24.8 to 1.26.3 upgrade, then inspect the operator's primary-election behavior using the reported log; done means the pod starts as primary in 1-master mode and accepts application connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.