Patroni health checks

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.6.2
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal
• Are you running Postgres Operator in production? yes
• Type of issue? question, feature request

Patroni has a watchdog process to detect cases in which patroni stops working. This helps in protecting the cluster against split brains

All my nodes have the following line in the logs:

WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"

I don't know if this is because the postgres-operator does not support watchdog or I have something misconfigured. I couldn't find any reference in the operator docs about watchdog.

I also noticed that the cluster pods do not have any health checks. If enabling the watchdog is not possible, maybe a health check to Patroni's REST API (8008) could replicate the same behaviour? This would be more k8s-friendly and avoid using kernel features

Start with the Patroni watchdog documentation linked in the issue and inspect how the operator configures cluster pods. Determine whether Linux watchdog support is expected or whether a health check against Patroni's REST API on port 8008 is the intended alternative. Done means the supported behavior is implemented or documented clearly, with the warning and health-check behavior addressed.