Postgres operator does not recover instance if K8S API temporarily unavailable

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.7.1
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Syseleven Metakube (Openshift)
• Are you running Postgres Operator in production? yes
• Type of issue? Bug report

We're using spilo image registry.opensource.zalan.do/acid/spilo-13:2.1-p1 and postgres version 13.

Unfortunately our manageg kubernetes provider has not the best availabilty for the k8s api.
If the api is not available temporarily, one, multiple or all patroni cluster members fail and can't recover.
If cluster status is checked with patronictl list the members of the cluster claiming "start failed".

The reason they can't (re)start seems to be, that somehow the pvc permissions get messed up, when the api problem happens:

2021-11-24 03:18:18 UTC [159287]: [1-1] 619daefa.26e37 0     FATAL:  data directory "/home/postgres/pgdata/pgroot/data" has invalid permissions
2021-11-24 03:18:18 UTC [159287]: [2-1] 619daefa.26e37 0     DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

If the instace is an replica you can fix the problem by patronictl reinit or restarting the pod.

Unfortunatly this can also happen for the leader. This can be fixed by restarting the pod.

When all 3 nodes become unready only a k -n namespcae delete po -l application=spilo --force --grace-period=0 can recover the cluster.

The main problem here is that the patroni cluster is not able to recover from such issue by itself.
If the leader is in "start failed" status, i would expect that one of the available replicas becomes the new master.

I also don't understand, why a not available K8S API leads to changes in the file permissions of the cluster member at all?

This is the complete postgres log of such case:

2021-11-24 03:17:00.804 31 LOG {ticks: 0, maint: 0, retry: 0}
2021-11-24 03:17:07,516 INFO: no action. I am (auth-postgres-db-1) the leader with the lock
2021-11-24 03:17:17,518 INFO: no action. I am (auth-postgres-db-1) the leader with the lock
2021-11-24 03:17:27,512 INFO: no action. I am (auth-postgres-db-1) the leader with the lock
2021-11-24 03:17:30.820 31 LOG {ticks: 0, maint: 0, retry: 0}
2021-11-24 03:17:38,183 INFO: no action. I am (auth-postgres-db-1) the leader with the lock
2021-11-24 03:17:47,515 INFO: no action. I am (auth-postgres-db-1) the leader with the lock
2021-11-24 03:17:57,519 INFO: no action. I am (auth-postgres-db-1) the leader with the lock
2021-11-24 03:17:59,854 ERROR: ObjectCache.run ProtocolError("Connection broken: TimeoutError(110, 'Connection timed out')", TimeoutError(110, 'Connection timed out'))
2021-11-24 03:18:00.820 31 LOG {ticks: 0, maint: 0, retry: 0}
2021-11-24 03:18:02,357 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416e544e0>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:04,862 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414abd518>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:18:05,863 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:09,003 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.10.10.1', port=443): Read timed out. (read timeout=2.5)",)': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:11,505 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416e546d8>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:18:12,507 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:15,011 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416e54358>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:17,498 ERROR: get_cluster
2021-11-24 03:18:17,500 ERROR: Error communicating with DCS
2021-11-24 03:18:17,515 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416e545c0>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:18:18,046 INFO: closed patroni connection to the postgresql cluster
2021-11-24 03:18:18,318 INFO: postmaster pid=159287
2021-11-24 03:18:18 UTC [159287]: [1-1] 619daefa.26e37 0     FATAL:  data directory "/home/postgres/pgdata/pgroot/data" has invalid permissions
2021-11-24 03:18:18 UTC [159287]: [2-1] 619daefa.26e37 0     DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
2021-11-24 03:18:18,517 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:19,347 ERROR: postmaster is not running
2021-11-24 03:18:19,348 INFO: demoted self because DCS is not accessible and i was a leader
2021-11-24 03:18:19,351 WARNING: Loop time exceeded, rescheduling immediately.
2021-11-24 03:18:21,022 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a27fd0>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:23,526 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a27f60>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:18:24,527 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:27,031 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a9bf28>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:29,351 ERROR: get_cluster
2021-11-24 03:18:29,352 ERROR: Error communicating with DCS
2021-11-24 03:18:29,353 INFO: DCS is not accessible
2021-11-24 03:18:29,354 WARNING: Loop time exceeded, rescheduling immediately.
2021-11-24 03:18:29,535 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a9bac8>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:18:30,537 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:30.843 31 LOG {ticks: 0, maint: 0, retry: 0}
2021-11-24 03:18:33,041 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416e1d5f8>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:36,551 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError('HTTPSConnectionPool(host=\'10.10.10.1\', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ReadTimeoutError("HTTPSConnectionPool(host=\'10.10.10.1\', port=443): Read timed out. (read timeout=2.5)",))',)
2021-11-24 03:18:37,552 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:39,354 ERROR: get_cluster
2021-11-24 03:18:39,355 ERROR: Error communicating with DCS
2021-11-24 03:18:39,355 INFO: DCS is not accessible
2021-11-24 03:18:39,356 WARNING: Loop time exceeded, rescheduling immediately.
2021-11-24 03:18:40,055 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416e1d4a8>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:42,559 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f04149f5128>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:18:43,561 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:46,996 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.10.10.1', port=443): Read timed out. (read timeout=2.5)",)': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:49,356 ERROR: get_cluster
2021-11-24 03:18:49,357 ERROR: Error communicating with DCS
2021-11-24 03:18:49,357 INFO: DCS is not accessible
2021-11-24 03:18:49,359 WARNING: Loop time exceeded, rescheduling immediately.
2021-11-24 03:18:49,500 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a68668>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:18:50,501 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:53,005 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a68048>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:55,506 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a68e10>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:18:56,508 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:18:59,010 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a68080>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db
2021-11-24 03:18:59,359 ERROR: get_cluster
2021-11-24 03:18:59,359 ERROR: Error communicating with DCS
2021-11-24 03:18:59,359 INFO: DCS is not accessible
2021-11-24 03:18:59,361 WARNING: Loop time exceeded, rescheduling immediately.
2021-11-24 03:19:00.834 31 ERROR connection error: PQconnectStart
2021-11-24 03:19:00.834 31 ERROR 
2021-11-24 03:19:00.834 31 ERROR connection error: PQconnectStart
2021-11-24 03:19:00.834 31 ERROR 
2021-11-24 03:19:00.834 31 WARNING postgres: default timeout
2021-11-24 03:19:00.834 31 ERROR connection error: PQconnectStart
2021-11-24 03:19:00.834 31 ERROR 
2021-11-24 03:19:00.834 31 WARNING hydra: default timeout
2021-11-24 03:19:00.846 31 LOG {ticks: 0, maint: 0, retry: 0}
2021-11-24 03:19:01,513 ERROR: Request to server https://10.10.10.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.10.10.1', port=443): Max retries exceeded with url: /api/v1/namespaces/auth/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dauth-postgres-db (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a68d30>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)'))",)
2021-11-24 03:19:02,515 ERROR: ObjectCache.run TypeError("unsupported operand type(s) for -: 'NoneType' and 'float'",)
2021-11-24 03:19:05,332 WARNING: Postgresql is not running.
2021-11-24 03:19:05,332 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:05,337 INFO: pg_controldata:
2021-11-24 03:19:05,337 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:05,357 INFO: Local timeline=1 lsn=3/27000028
2021-11-24 03:19:05,372 INFO: master_timeline=2
2021-11-24 03:19:05,377 INFO: master: history=1	3/270000A0	no recovery target specified
2021-11-24 03:19:05,378 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:07,881 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f04155377f0>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/pods/auth-postgres-db-1
2021-11-24 03:19:10,385 INFO: starting as a secondary
2021-11-24 03:19:10,388 WARNING: Loop time exceeded, rescheduling immediately.
2021-11-24 03:19:10 UTC [159304]: [1-1] 619daf2e.26e48 0     FATAL:  data directory "/home/postgres/pgdata/pgroot/data" has invalid permissions
2021-11-24 03:19:10 UTC [159304]: [2-1] 619daf2e.26e48 0     DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
2021-11-24 03:19:10,708 INFO: postmaster pid=159304
2021-11-24 03:19:10,390 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:12,896 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416e07a58>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/pods/auth-postgres-db-1
2021-11-24 03:19:14,594 WARNING: Retry got exception: 'connection problems'
2021-11-24 03:19:15,399 INFO: restarting after failure in progress
2021-11-24 03:19:15,416 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:17,921 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414abda58>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/pods/auth-postgres-db-1
2021-11-24 03:19:20,425 INFO: failed to start postgres
2021-11-24 03:19:20,426 WARNING: Postgresql is not running.
2021-11-24 03:19:20,426 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:20,430 INFO: pg_controldata:
2021-11-24 03:19:20,431 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:20,449 INFO: Local timeline=1 lsn=3/27000028
2021-11-24 03:19:20,461 INFO: master_timeline=2
2021-11-24 03:19:20,466 INFO: master: history=1	3/270000A0	no recovery target specified
2021-11-24 03:19:20,467 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:22,971 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414a2ae10>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/pods/auth-postgres-db-1
2021-11-24 03:19:25,472 INFO: starting as a secondary
2021-11-24 03:19:25 UTC [159318]: [1-1] 619daf3d.26e56 0     FATAL:  data directory "/home/postgres/pgdata/pgroot/data" has invalid permissions
2021-11-24 03:19:25 UTC [159318]: [2-1] 619daf3d.26e56 0     DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
2021-11-24 03:19:25,714 INFO: postmaster pid=159318
2021-11-24 03:19:25,743 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:28,246 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0414ada7f0>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/pods/auth-postgres-db-1
2021-11-24 03:19:30,750 INFO: failed to start postgres
2021-11-24 03:19:30.857 31 LOG {ticks: 0, maint: 0, retry: 0}
2021-11-24 03:19:31,224 WARNING: Postgresql is not running.
2021-11-24 03:19:31,224 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:31,229 INFO: pg_controldata:
2021-11-24 03:19:31,229 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:31,246 INFO: Local timeline=1 lsn=3/27000028
2021-11-24 03:19:31,256 INFO: master_timeline=2
2021-11-24 03:19:31,260 INFO: master: history=1	3/270000A0	no recovery target specified
2021-11-24 03:19:31,260 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:33,764 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416dd3860>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/pods/auth-postgres-db-1
2021-11-24 03:19:36,268 INFO: starting as a secondary
2021-11-24 03:19:36 UTC [159331]: [1-1] 619daf48.26e63 0     FATAL:  data directory "/home/postgres/pgdata/pgroot/data" has invalid permissions
2021-11-24 03:19:36 UTC [159331]: [2-1] 619daf48.26e63 0     DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
2021-11-24 03:19:36,512 INFO: postmaster pid=159331
2021-11-24 03:19:36,526 INFO: Lock owner: auth-postgres-db-0; I am auth-postgres-db-1
2021-11-24 03:19:39,031 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0416dcb390>, 'Connection to 10.10.10.1 timed out. (connect timeout=2.5)')': /api/v1/namespaces/auth/pods/auth-postgres-db-1
2021-11-24 03:19:41,534 INFO: failed to start postgres
2021-11-24 03:19:41,535 WARNING: Postgresql is not running.

..... repeating unti restart...

Start with the operator's Kubernetes/DCS recovery behavior and the supplied Patroni logs, using patronictl list to observe the start failed state during a temporary API outage. Done means cluster members recover without manual pod deletion or patronictl reinit, while preserving the documented leader and replica behavior.