Cluster failed to progress beyond "initialization" after initialization and corruption

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.6.0
  registry.opensource.zalan.do/acid/postgres-operator:v1.6.0

• Where do you run it - cloud or metal? Kubernetes or OpenShift? [AWS K8s | GCP ... | Bare Metal K8s]
  k3s

• Are you running Postgres Operator in production? [yes | no]
  not really

• Type of issue? [Bug report, question, feature request, etc.]
  bug report

Steps to reproduce:

1. Create a cluster
2. Have a catastrophic event that potentially corrupts Patroni data volume (mine was OpenEBS volume attaching crashing cascades)
3. Try to restart the cluster
4. See a lot of these lines in the log

2021-01-19 18:19:19,751 INFO: waiting for leader to bootstrap
2021-01-19 18:19:29,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:19:29,751 INFO: waiting for leader to bootstrap
2021-01-19 18:19:39,751 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:19:39,751 INFO: waiting for leader to bootstrap
2021-01-19 18:19:49,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:19:49,750 INFO: waiting for leader to bootstrap
2021-01-19 18:19:59,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:19:59,750 INFO: waiting for leader to bootstrap
2021-01-19 18:20:09,751 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:20:09,751 INFO: waiting for leader to bootstrap
2021-01-19 18:20:19,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:20:19,751 INFO: waiting for leader to bootstrap
2021-01-19 18:20:29,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:20:29,751 INFO: waiting for leader to bootstrap
2021-01-19 18:20:39,751 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:20:39,751 INFO: waiting for leader to bootstrap
2021-01-19 18:20:49,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:20:49,750 INFO: waiting for leader to bootstrap
2021-01-19 18:20:59,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:20:59,750 INFO: waiting for leader to bootstrap
2021-01-19 18:21:09,751 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:21:09,751 INFO: waiting for leader to bootstrap
2021-01-19 18:21:19,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:21:19,751 INFO: waiting for leader to bootstrap
2021-01-19 18:21:29,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:21:29,750 INFO: waiting for leader to bootstrap
2021-01-19 18:21:39,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:21:39,751 INFO: waiting for leader to bootstrap
2021-01-19 18:21:49,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:21:49,751 INFO: waiting for leader to bootstrap
2021-01-19 18:21:59,750 INFO: Lock owner: None; I am acid-test-0
2021-01-19 18:21:59,750 INFO: waiting for leader to bootstrap    

...and it keeps going ad infinitum. Adding more or decreasing the number of cluster instances won't help.

My speculation is that because Patroni now sit in the veil between life and death, all because there is some corruptions when I tried to repair all the damage my network outage caused (which caused OpenEBS to fail, so I have to do fsck manually on all PV[C]s). Since this is a test database and I haven't really made any backups yet (and for some reason backups didn't work as expected cause I can't even open the backup tab in the UI), but I still want to rescue it because from what I observed, the pgdata behind it is perfectly intact.

I've seen from an article that removing an etcd key would help bypass the "initialization" process. This however makes me wonder if the Patronl under the k8s operator would ever had used the internal etcd at all of what the k8s master nodes are using.

Start from the cluster restart sequence and the repeated Patroni “waiting for leader to bootstrap” logs after the volume corruption event. Inspect how the operator handles corrupted Patroni data during initialization in the reported k3s deployment. Done means the cluster can recover or clearly report the failure instead of remaining indefinitely in initialization.