Logical backup faillure

Hi,

I have some issues to setup my backup.
Logs from the cronjob:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8315    0  8315    0     0   902k      0 --:--:-- --:--:-- --:--:-- 1015k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   165  100   165    0     0   5500      0 --:--:-- --:--:-- --:--:--  5500
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 28313    0 28313    0     0  1382k      0 --:--:-- --:--:-- --:--:-- 1455k
+ dump
+ /usr/lib/postgresql/12/bin/pg_dumpall
+ compress
+ pigz
++ estimate_size
++ /usr/lib/postgresql/12/bin/psql -tqAc 'select sum(pg_database_size(datname)::numeric) from pg_database;'
psql: error: could not connect to server: FATAL:  database "pg-root" does not exist
/dump.sh: line 96: / DUMP_SIZE_COEFF: syntax error: operand expected (error token is "/ DUMP_SIZE_COEFF")
pg_dumpall: error: pg_dump failed on database "hivebrite", exiting

Then I tried to setup a pod o troubleshoot the issue with the following manifest:

apiVersion: v1
kind: Pod
metadata:
  labels:
    application: spilo-logical-backup
    cluster-name: acid-pgop-postgres-alumni-connect
  name: logical-backup-acid-pgop-postgres-alumni-connect-damien
  namespace: default
spec:
  affinity:
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              application: spilo-logical-backup
              cluster-name: acid-pgop-postgres-alumni-connect
          topologyKey: kubernetes.io/hostname
        weight: 1
  containers:
  - env:
    - name: SCOPE
      value: acid-pgop-postgres-alumni-connect
    - name: CLUSTER_NAME_LABEL
      value: cluster-name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: LOGICAL_BACKUP_S3_BUCKET
      value: hivebrite-backup-pgop-integration
    - name: LOGICAL_BACKUP_S3_REGION
      value: eu-west-1
    - name: LOGICAL_BACKUP_S3_ENDPOINT
    - name: LOGICAL_BACKUP_S3_SSE
      value: AES256
    - name: LOGICAL_BACKUP_S3_BUCKET_SCOPE_SUFFIX
      value: /ce8ddfcd-e779-455c-ba32-80e29615568b
    - name: PG_VERSION
      value: "12"
    - name: PGPORT
      value: "5432"
    - name: PGUSER
      value: pg-root
    - name: PGDATABASE
      value: pg-root
    - name: PGSSLMODE
      value: require
    - name: PGPASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: pg-root.acid-pgop-postgres-alumni-connect.credentials
    - name: AWS_ACCESS_KEY_ID
      value: xxxxxxxxxxxxxxxxxxxxxxxxx
    - name: AWS_SECRET_ACCESS_KEY
      value: xxxxxxxxxxxxxxxxxxxxxxxxx
    image: registry.opensource.zalan.do/acid/logical-backup:master-58
    imagePullPolicy: IfNotPresent
    name: logical-backup
    ports:
    - containerPort: 8008
      protocol: TCP
    - containerPort: 5432
      protocol: TCP
    - containerPort: 8080
      protocol: TCP
    resources:
      limits:
        cpu: "3"
        memory: 7372Mi
      requests:
        cpu: "3"
        memory: 7372Mi
    command: ["sleep", "1d"]
    securityContext:
      privileged: false
      readOnlyRootFilesystem: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: postgres-pod-token-mtdxp
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: gke-integration-integration-general-p-d5d6ef4a-83j7
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: postgres-pod
  serviceAccountName: postgres-pod
  terminationGracePeriodSeconds: 300
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: postgres-pod-token-mtdxp
    secret:
      defaultMode: 420
      secretName: postgres-pod-token-mtdxp

In this pod I tried to run the script step by step.
The issue seems to be on this line: https://github.com/zalando/postgres-operator/blob/master/docker/logical-backup/dump.sh#L96

So I tried to only call the dump function, it works.
Then I tried: dump | compress, there is an issue

+ dump
+ compress
+ pigz
+ /usr/lib/postgresql/12/bin/pg_dumpall
Usage: pigz [options] [files ...]
  will compress files in place, adding the suffix '.gz'. If no files are
  specified, stdin will be compressed to stdout. pigz does what gzip does,
  but spreads the work over multiple processors and cores when compressing.

Options:
  -0 to -9, -11        Compression level (level 11, zopfli, is much slower)
  --fast, --best       Compression levels 1 and 9 respectively
  -b, --blocksize mmm  Set compression block size to mmmK (default 128K)
  -c, --stdout         Write all processed output to stdout (won't delete)
  -d, --decompress     Decompress the compressed input
  -f, --force          Force overwrite, compress .gz, links, and to terminal
  -F  --first          Do iterations first, before block split for -11
  -h, --help           Display a help screen and quit
  -i, --independent    Compress blocks independently for damage recovery
  -I, --iterations n   Number of iterations for -11 optimization
  -J, --maxsplits n    Maximum number of split blocks for -11
  -k, --keep           Do not delete original file after processing
  -K, --zip            Compress to PKWare zip (.zip) single entry format
  -l, --list           List the contents of the compressed input
  -L, --license        Display the pigz license and quit
  -m, --no-time        Do not store or restore mod time
  -M, --time           Store or restore mod time
  -n, --no-name        Do not store or restore file name or mod time
  -N, --name           Store or restore file name and mod time
  -O  --oneblock       Do not split into smaller blocks for -11
  -p, --processes n    Allow up to n compression threads (default is the
                       number of online processors, or 8 if unknown)
  -q, --quiet          Print no messages, even on error
  -r, --recursive      Process the contents of all subdirectories
  -R, --rsyncable      Input-determined block locations for rsync
  -S, --suffix .sss    Use suffix .sss instead of .gz (for compression)
  -t, --test           Test the integrity of the compressed input
  -v, --verbose        Provide more verbose output
  -V  --version        Show the version of pigz
  -Y  --synchronous    Force output file write to permanent storage
  -z, --zlib           Compress to zlib (.zz) instead of gzip format
  --                   All arguments after "--" are treated as file

I tried also: dump | aws_upload $(($(estimate_size) / DUMP_SIZE_COEFF))

+ dump
+ /usr/lib/postgresql/12/bin/pg_dumpall
++ estimate_size
++ /usr/lib/postgresql/12/bin/psql -tqAc 'select sum(pg_database_size(datname)::numeric) from pg_database;'
psql: error: could not connect to server: FATAL:  database "pg-root" does not exist
bash: / DUMP_SIZE_COEFF: syntax error: operand expected (error token is "/ DUMP_SIZE_COEFF")

At the final I tried again the full piped commands: dump | compress | aws_upload $(($(estimate_size) / DUMP_SIZE_COEFF))

+ compress
+ pigz
+ dump
+ /usr/lib/postgresql/12/bin/pg_dumpall
++ estimate_size
++ /usr/lib/postgresql/12/bin/psql -tqAc 'select sum(pg_database_size(datname)::numeric) from pg_database;'
psql: error: could not connect to server: FATAL:  database "pg-root" does not exist
bash: / DUMP_SIZE_COEFF: syntax error: operand expected (error token is "/ DUMP_SIZE_COEFF")
pg_dumpall: error: pg_dump failed on database "hivebrite", exiting

So I can identify 2 issues:

• I seems normally I should have a pg-root database in my postgres
• For an unknown reason (from my point of view), the dump for ma database hivebrite fail

I'm running with these versions:
postgres: 12
operator: 1.5.0
backup image: registry.opensource.zalan.do/acid/logical-backup:master-58

Start by reading docker/logical-backup/dump.sh around line 96 and rerun the logged dump, compress, and estimate_size pipeline in the supplied pod manifest. Done means identifying why the pg-root connection and hivebrite dump fail and fixing the backup command so the logical backup completes.