zalando / zalando/postgres-operator

Adjustable Timeout Needed for Leader Election in PostgreSQL Operator

Open
#2,887 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

  • Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.13.0
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s on rke2
  • Are you running Postgres Operator in production? yes
  • Type of issue? Bug report/feature request

We’ve observed that when etcd is under heavy load, the PostgreSQL operator fails to complete the setup of a database cluster. Based on my understanding of the code, the operator attempts to communicate with etcd five times in quick succession to designate a leader pod and initiate the database startup. However, in scenarios where we were using slower Azure disks—combined with etcd being under load—the new PostgreSQL database pod became stuck in the leader election process and never recovered.

Is there a way to increase the timeout in the operator to handle such cases? If not can it be added?
Generally, we do not understand why leader election is retry limited.
We only encounter slow disks in our test environments.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the operator's leader-election communication with etcd and its five retry attempts in the v1.13.0 behavior described here. Determine how a configurable timeout or retry policy could cover slow disks and overloaded etcd; done means a stuck database pod can recover or wait according to the configured behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, distributed-systems, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.