zalando / zalando/postgres-operator

Replica in archive recovery mode

Open
#2,964 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discussion enhancement
Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

This morning, for the first time, and an unnown reason, I saw this log:

2025-10-16 06:01:10,984 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:01:20,972 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:01:30,977 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:01:41,011 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:01:50,975 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:02:01,053 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:02:11,008 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:02:20,977 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:02:30,978 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:02:40,982 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:02:51,482 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:03:01,482 INFO: no action. I am (entre-vos-mains--de-pg-0), a secondary, and following a leader (entre-vos-mains--de-pg-1)
2025-10-16 06:03:11,487 INFO: Got response from entre-vos-mains--de-pg-1 http://100.64.24.113:8008/patroni: {"state": "running", "postmaster_start_time": "2025-10-09 05:03:13.898519+00:00", "role": "master", "server_version": 150010, "xlog": {"location": 487311174984}, "timeline": 52, "replication": [{"usename": "standby", "application_name": "entre-vos-mains--de-pg-0", "client_addr": "100.64.29.244", "state": "streaming", "sync_state": "async", "sync_priority": 0}], "dcs_last_seen": 1760594590, "database_system_identifier": "7276070543149191243", "patroni": {"version": "3.3.3", "scope": "entre-vos-mains--de-pg", "name": "entre-vos-mains--de-pg-1"}}
2025-10-16 06:03:11,489 WARNING: Primary (entre-vos-mains--de-pg-1) is still alive
server signaled
2025-10-16 06:03:11,500 INFO: following a different leader because i am not the healthiest node
2025-10-16 06:03:20,987 INFO: Got response from entre-vos-mains--de-pg-1 http://100.64.24.113:8008/patroni: {"state": "running", "postmaster_start_time": "2025-10-09 05:03:13.898519+00:00", "role": "master", "server_version": 150010, "xlog": {"location": 487311174984}, "timeline": 52, "dcs_last_seen": 1760594600, "database_system_identifier": "7276070543149191243", "patroni": {"version": "3.3.3", "scope": "entre-vos-mains--de-pg", "name": "entre-vos-mains--de-pg-1"}}
2025-10-16 06:03:20,988 WARNING: Primary (entre-vos-mains--de-pg-1) is still alive
2025-10-16 06:03:21,041 INFO: following a different leader because i am not the healthiest node
2025-10-16 06:03:30,985 INFO: Got response from entre-vos-mains--de-pg-1 http://100.64.24.113:8008/patroni: {"state": "running", "postmaster_start_time": "2025-10-09 05:03:13.898519+00:00", "role": "master", "server_version": 150010, "xlog": {"location": 487311252208}, "timeline": 52, "dcs_last_seen": 1760594610, "database_system_identifier": "7276070543149191243", "patroni": {"version": "3.3.3", "scope": "entre-vos-mains--de-pg", "name": "entre-vos-mains--de-pg-1"}}

Patronictl list gave for the replica:

in archive recovery

I wonder what happened.

restarting the pod worked.

I noticed it with increase cpu usage on our object storage ingress.

I think it would be nice to show the CRD status to something like degraded in this case.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the operator observes Patroni's "in archive recovery" state and updates the cluster CRD status; the issue names no source files or tests. Reproduce or inspect the reported log sequence, then determine what status should be exposed when a replica remains in that state and add coverage for the observed transition.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.