zalando / zalando/postgres-operator

Bug in failsafe_mode ?

Open
#3,055 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.14.0
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? DigitalOcean managed K8S
  • Are you running Postgres Operator in production? yes
  • Type of issue? Bug report

Leader replica was failed to connect to K8S API at but cluster maintained its state due to failsafe_mode enabled.

2026-03-18 15:21:36,310 ERROR: Error communicating with DCS
2026-03-18 15:21:36,314 INFO: Got response from develop-postgresql-0 http://10.244.4.163:8008/failsafe: Accepted
2026-03-18 15:21:36,315 INFO: continue to run as a leader because failsafe mode is enabled and all members are accessible

It works as expected.

But after 30 seconds master gone into Demoting self (immediate-nolock) mode. I see no error regarding "got no response from https://X.X.X.X/failsafe", leader just decided to demote itself.

Reading https://patroni.readthedocs.io/en/master/dcs_failsafe_mode.html I assume cluster should run indefinitely while other replicas available. From my perspective it looks like an bug in operator.

2026-03-18 15:21:36,316 WARNING: Loop time exceeded, rescheduling immediately.
2026-03-18 15:21:37.788 UTC [29] LOG {ticks: 0, maint: 0, retry: 0}
2026-03-18 15:21:38,416 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/production/endpoints/develop-postgresql
2026-03-18 15:21:38,433 INFO: Could not take out TTL lock
2026-03-18 15:21:38,434 INFO: Demoting self (immediate-nolock)

More log lines - https://gist.github.com/baznikin/608ff8709f00a008723080e4388a579e

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failsafe_mode behavior described in the Patroni documentation and compare it with the Kubernetes API, TTL-lock, and demotion log sequence in the issue and linked gist. Determine why the leader demotes after the accepted failsafe response; done means the cause is identified and the operator's behavior matches the documented failsafe expectation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.