Backup fails when sidecars are in use

When you're using sidecars that use iptables to intercept traffic, the logical backup is failing. Normally, I can configure an operator with a k8s section that overrides the args to wait on the Istio sidecar proxy to be up and responsive, but I haven't found a way except rebuilding the image, to do that here. This means the logical backups fail when trying to contact the k8s API server (connection refused), and there are no retries implemented (which would be another solution).

Solutions thus could be;

1. enable randomised exponential backoff for all network requests from the backup container to automatically wait for the sidecar to be up and running
2. enable the operator to merge in (override) bits of the job spec in the operator configuration manifest, so I can prepend a loop that awaits network connectivity, like so:

        command: ["sh", "-c"]
        args:
        - |
          set -e
          trap "curl --max-time 2 -s -f -XPOST http://127.0.0.1:15000/quitquitquit" EXIT
          while ! curl -s -f http://127.0.0.1:15020/healthz/ready; do sleep 1; done
          exec /dump.sh

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.5.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? GCP
• Are you running Postgres Operator in production? yes
• Type of issue? bug report / feature

$ k logs logical-backup-app-analytics-db-1609633800-586qh  -c logical-backup
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to 172.16.16.1 port 443: Connection refused

Refs

• https://docs.timescale.com/latest/using-timescaledb/backup#docker-wale

Start by tracing the logical-backup container and the operator-generated backup Job specification; the report identifies the failing Kubernetes API connection and shows a possible command/args wrapper. Reproduce the sidecar startup race with the provided log, then determine whether the selected retry or job-spec override behavior is supported. Done means logical backups reliably reach the Kubernetes API when an Istio sidecar is present.