Postgres-operator severed from cluster by changing `postgres.*.credentials` secret

Hi,

Changing postgres user password by modifying Kubernetes postgres.*.credentials Secret causes error in postgres-operator:

time="2019-03-27T18:51:01Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: pq: password authentication failed for user \"postgres\"" cluster-name=postgres-operator-pg-test/infra-postgres-operator-test-patroni pkg=cluster
time="2019-03-27T18:51:01Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: pq: password authentication failed for user \"postgres\"" cluster-name=postgres-operator-pg-test/infra-postgres-operator-test-patroni pkg=controller worker=0

The postgres-operator cannot connect to the Postgres because it's trying to use the new password (from Secret) and fails.

Possible solutions:

• port-forward so the connection will be authenticated by using trusted IP address in pg_hba.conf
• extend Patroni REST API to be able to change superuser password after initialisation
• exec into master spilo Pod and run psql from there (yuk!)

Start with the cluster sync and role-connection path described in the error, then inspect how the Kubernetes credentials Secret is consumed. Compare the listed port-forward, Patroni REST API, and Spilo Pod approaches. Done means changing the postgres password no longer prevents the operator from reconnecting and syncing roles.