zalando / zalando/postgres-operator

Logical backup job fails if pooler is enabled for the cluster - pg_dump: error: connection to database "template1"

Open
#1,411 8 comments 10 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using?
    registry.opensource.zalan.do/acid/postgres-operator:v1.6.1
  • **Where do you run it - cloud or metal?
    Kubernetes with kops on AWS
  • Are you running Postgres Operator in production?
    we are going to
  • Type of issue?
    Bug report

When the cluster manifest enables poolers cronjobs for logical backups fail with error pg_dump: error: connection to database "template1" failed: ERROR: unexpected response from login query

job full logs:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12220    0 12220    0     0   917k      0 --:--:-- --:--:-- --:--:--  994k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20702    0 20702    0     0  1555k      0 --:--:-- --:--:-- --:--:-- 1555k
+ dump
+ /usr/lib/postgresql/13/bin/pg_dumpall
+ compress
+ pigz
+ upload
+ case $LOGICAL_BACKUP_PROVIDER in
++ estimate_size
++ /usr/lib/postgresql/13/bin/psql -tqAc 'select sum(pg_database_size(datname)::numeric) from pg_database;'
+ aws_upload 6710488
+ declare -r EXPECTED_SIZE=6710488
++ date +%s
+ PATH_TO_BACKUP=s3://postgres-operator-backups-bucket/spilo/test-db/aae491ea-8a1b-4d1f-b5bd-755466eae34c/logical_backups/1616113817.sql.gz
+ args=()
+ [[ ! -z 6710488 ]]
+ args+=("--expected-size=$EXPECTED_SIZE")
+ [[ ! -z '' ]]
+ [[ ! -z us-east-1 ]]
+ args+=("--region=$LOGICAL_BACKUP_S3_REGION")
+ [[ ! -z AES256 ]]
+ args+=("--sse=$LOGICAL_BACKUP_S3_SSE")
+ aws s3 cp - s3://postgres-operator-backups-bucket/spilo/test-db/aae491ea-8a1b-4d1f-b5bd-755466eae34c/logical_backups/1616113817.sql.gz --expected-size=6710488 --region=us-east-1 --sse=AES256
pg_dump: error: connection to database "template1" failed: ERROR:  unexpected response from login query
pg_dumpall: error: pg_dump failed on database "template1", exiting

cluster manifest:

apiVersion: "acid.zalan.do/v1"
kind: "postgresql"
metadata:
  name: "test-db"
  namespace: "postgres-test"
  labels:
    team: test
spec:
  teamId: "test"
  postgresql:
    version: "13"
    parameters:
      max_connections: "100"
  numberOfInstances: 2
  enableConnectionPooler: true
  enableReplicaConnectionPooler: true
  enableLogicalBackup: true
  volume:
    size: "5Gi"
  users:
    admin:
    - superuser
    - createdb
    test-user: []
  databases:
    test: test-user
  resources:
    requests:
      cpu: 100m
      memory: 500Mi
    limits:
      cpu: 1000m
      memory: 2000Mi
  sidecars:
  - name: postgres-exporter
    image: wrouesnel/postgres_exporter
    env:
    - name: DATA_SOURCE_USER
      valueFrom:
        secretKeyRef:
          name: postgres.test-db.credentials.postgresql.acid.zalan.do
          key: username
    - name: DATA_SOURCE_PASS
      valueFrom:
        secretKeyRef:
          key: password
          name: postgres.test-db.credentials.postgresql.acid.zalan.do
    - name: "DATA_SOURCE_NAME"
      value: "postgresql://$(DATA_SOURCE_USER):$(DATA_SOURCE_PASS)@$(POD_NAME):5432/postgres"
    ports:
    - containerPort: 9187
      protocol: TCP
    resources:
      limits:
        cpu: 500m
        memory: 100Mi
      requests:
        cpu: 100m
        memory: 100Mi

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the logical-backup job's pg_dumpall invocation and the manifest settings enableConnectionPooler, enableReplicaConnectionPooler, and enableLogicalBackup; reproduce the template1 failure with the supplied Kubernetes configuration. Done means logical backups complete successfully when poolers are enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.