Cannot start a standby cluster (Patroni incorrectly promotes itself as master and stops replicating)

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.5.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? [Bare Metal K8s (RKE on a bunch of OCI VMs)]
• Are you running Postgres Operator in production? [no, because this is our very first experiment, so we'll move it to production as soon as this test succeeds!]
• Type of issue? [Bug report]

Cluster log:

2020-11-16 16:15:53,019 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2020-11-16 16:15:53,031 - bootstrapping - INFO - Looks like your running openstack
2020-11-16 16:15:53,082 - bootstrapping - INFO - Configuring pam-oauth2
2020-11-16 16:15:53,082 - bootstrapping - INFO - No PAM_OAUTH2 configuration was specified, skipping
2020-11-16 16:15:53,083 - bootstrapping - INFO - Configuring wal-e
2020-11-16 16:15:53,083 - bootstrapping - INFO - Configuring pgbouncer
2020-11-16 16:15:53,083 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2020-11-16 16:15:53,083 - bootstrapping - INFO - Configuring pgqd
2020-11-16 16:15:53,083 - bootstrapping - INFO - Configuring crontab
2020-11-16 16:15:53,083 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2020-11-16 16:15:53,083 - bootstrapping - INFO - Configuring patroni
2020-11-16 16:15:53,092 - bootstrapping - INFO - Writing to file /home/postgres/postgres.yml
2020-11-16 16:15:53,092 - bootstrapping - INFO - Configuring bootstrap
2020-11-16 16:15:53,093 - bootstrapping - INFO - Writing to file /home/postgres/.pgpass_timescaledb
2020-11-16 16:15:53,093 - bootstrapping - INFO - Configuring certificate
2020-11-16 16:15:53,093 - bootstrapping - INFO - Generating ssl certificate
2020-11-16 16:15:53,115 - bootstrapping - INFO - Configuring log
2020-11-16 16:15:53,116 - bootstrapping - INFO - Configuring standby-cluster
2020-11-16 16:15:54,394 INFO: No PostgreSQL configuration items changed, nothing to reload.
2020-11-16 16:15:54,396 INFO: Lock owner: None; I am exacto-timescaledb-0
2020-11-16 16:15:54,425 INFO: trying to bootstrap a new cluster
2020-11-16 16:15:54,426 INFO: Running custom bootstrap script: python3 /scripts/clone_with_basebackup.py --pgpass=/home/postgres/.pgpass_timescaledb --host=timescaledb --port=5432 --user="standby"
2020-11-16 16:15:54,473 INFO: cloning cluster exacto-timescaledb from "host=timescaledb port=5432 user=standby dbname=postgres"
2020-11-16 16:16:04,897 INFO: Lock owner: None; I am exacto-timescaledb-0
2020-11-16 16:16:04,897 INFO: not healthy enough for leader race
2020-11-16 16:16:04,906 INFO: bootstrap in progress

Last 3 rows repeat identical every 10 seconds, then a different row:

2020-11-16 16:15:54,394 INFO: No PostgreSQL configuration items changed, nothing to reload.

Then the same 3 rows every 10 seconds for approx 9 minutes, and finally:

**> 2020-11-16 16:29:03 pg_basebackup: error: could not read COPY data: server closed the connection unexpectedly

2020-11-16 16:29:03 This probably means the server terminated abnormally
2020-11-16 16:29:03 before or while processing the request.
2020-11-16 16:29:03 pg_basebackup: removing contents of data directory "/home/postgres/pgdata/pgroot/data"

At the same time, the log of the source Postgres cluster ("timescaledb") says:

2020-11-16 16:29:03.474 UTC [3223] LOG: could not send data to client: Connection reset by peer
2020-11-16 16:29:03.474 UTC [3223] ERROR: base backup could not send data, aborting backup
2020-11-16 16:29:03.474 UTC [3223] LOG: could not send data to client: Broken pipe
2020-11-16 16:29:03.474 UTC [3223] FATAL: connection to client lost
2020-11-16 16:29:06.420 UTC [3235] LOG: could not receive data from client: Connection reset by peer
2020-11-16 16:29:06.420 UTC [3235] LOG: unexpected EOF on standby connection**

Back to the failing standby cluster:

2020-11-16 16:29:06,420 ERROR: Clone failed
2020-11-16 16:29:06 Traceback (most recent call last):
2020-11-16 16:29:06 File "/scripts/clone_with_basebackup.py", line 61, in main
2020-11-16 16:29:06 run_basebackup(options)
2020-11-16 16:29:06 File "/scripts/clone_with_basebackup.py", line 54, in run_basebackup
2020-11-16 16:29:06 raise Exception("pg_basebackup exited with code={0}".format(ret))
2020-11-16 16:29:06 Exception: pg_basebackup exited with code=1
2020-11-16 16:29:06,431 INFO: removing initialize key after failed attempt to bootstrap the cluster
2020-11-16 16:29:06,479 INFO: renaming data directory to /home/postgres/pgdata/pgroot/data_2020-11-16-16-29-06
2020-11-16 16:29:06,494 ERROR: Unexpected exception
2020-11-16 16:29:06 Traceback (most recent call last):
2020-11-16 16:29:06 File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1385, in run_cycle
2020-11-16 16:29:06 info = self._run_cycle()
2020-11-16 16:29:06 File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1288, in _run_cycle
2020-11-16 16:29:06 return self.post_bootstrap()
2020-11-16 16:29:06 File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1184, in post_bootstrap
2020-11-16 16:29:06 self.cancel_initialization()
2020-11-16 16:29:06 File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1179, in cancel_initialization
2020-11-16 16:29:06 raise PatroniException('Failed to bootstrap cluster')
2020-11-16 16:29:06 patroni.exceptions.PatroniException: 'Failed to bootstrap cluster'
2020-11-16 16:29:06,495 INFO: Unexpected exception raised, please report it as a BUG

And last 13 rows repeat infinitely every 10 seconds, so I've been to extract these logs thanks to my Grafana/Loki stack, and would I have been left without any logs otherways.

Source and target clusters seem to accuse each other for the connection closure...
Please advise, as we need to finalize this first test in order to be able to move postgres-operator to Production as soon as possible!
Thank you for this admirable piece of software!

Start with the reported standby-cluster logs and reproduce the failure using the PostgreSQL Operator image registry.opensource.zalan.do/acid/postgres-operator:v1.5.0. Read /scripts/clone_with_basebackup.py and the referenced Patroni paths in patroni/ha.py, then trace the base-backup connection and bootstrap retry behavior. Done means the standby completes cloning, continues replicating, and does not promote itself unexpectedly.