K8s: Restart in the middle of reinit of replica db node -> manual fixing of db node needed

Image: registry.opensource.zalan.do/acid/postgres-operator:v1.6.0
patroni==2.0.1
spilo-13:2.0-p4
PostgreSQL 13.2

Running it in Kubernetes
Running Postgres operator in Test pipe Environment
Issue: Feature request

Main Error: Data dir for the cluster is not empty, but system ID is invalid; consider doing reinitialize

There was 20 million rows in the DB cluster.

Following is quite a simple test execution.
Steps,
reinit:

Are you sure you want to reinitialize members rdbms-pg-cluster-2? [y/N]: y
Success: reinitialize for member rdbms-pg-cluster-2

Then the same replica pod was restarted in the middle of the reinit process:

kubectl delete po rdbms-pg-cluster-2
pod "rdbms-pg-cluster-2" deleted

Then the pod got in mentioned error state:

"data dir for the cluster is not empty, but system ID is invalid; consider doing reinitialize"

Such needs manual fixing.

Larger the database → Higher the probability for these kinds of problems.

Patroni REST interface of the problematic DB pod / DB pod works at that time:

curl --max-time 10 "http://xxxx.xx.xx.xx:8008/cluster"
{"members": ... {"name": "rdbms-pg-cluster-2", "role": "replica", "state": "stopped", "api_url": "http://xx.xx.xx.xx:8008/patroni", "host": "xxx.xx.xx.xx", "port": 5432, "lag": "unknown"}]}

And it is possible to reinitialize that pod with curl:

curl --max-time 10 -s -X POST "http://xxx.xx.xx.xx:8008/reinitialize"

So, in theory, it would be possible to write our own script for automatic re initialization for some cases. But is it wise, I do not know.

Q: What could be done for this problem?

I think, in source codes of Patroni, the error message comes from following place:

(https://github.com/zalando/patroni/blob/master/patroni/ha.py)

if not self.sysid_valid(data_sysid):
# data directory is not empty, but no valid sysid, cluster must be broken, suggest reinit
return ("data dir for the cluster is not empty, "
"but system ID is invalid; consider doing reinitialize")

Attached Logs, CRD and manifest files here for reference :
reinit-error.txt
manifest.txt
crd.txt

remove_data_directory_on_diverged_timelines: 'true'
remove_data_directory_on_rewind_failure: 'true'

Start with the attached reinit-error.txt, manifest.txt, and crd.txt, then inspect the referenced Patroni entry point in patroni/ha.py. Determine whether the operator should handle a replica restart during reinitialization and document the safe completion criteria for any proposed recovery behavior.