zalando / zalando/postgres-operator
Logical backup job fails if pooler is enabled for the cluster - pg_dump: error: connection to database "template1"
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
Please, answer some short questions which should help us to understand your problem / question better?
- Which image of the operator are you using?
registry.opensource.zalan.do/acid/postgres-operator:v1.6.1 - **Where do you run it - cloud or metal?
Kubernetes with kops on AWS - Are you running Postgres Operator in production?
we are going to - Type of issue?
Bug report
When the cluster manifest enables poolers cronjobs for logical backups fail with error pg_dump: error: connection to database "template1" failed: ERROR: unexpected response from login query
job full logs:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12220 0 12220 0 0 917k 0 --:--:-- --:--:-- --:--:-- 994k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20702 0 20702 0 0 1555k 0 --:--:-- --:--:-- --:--:-- 1555k
+ dump
+ /usr/lib/postgresql/13/bin/pg_dumpall
+ compress
+ pigz
+ upload
+ case $LOGICAL_BACKUP_PROVIDER in
++ estimate_size
++ /usr/lib/postgresql/13/bin/psql -tqAc 'select sum(pg_database_size(datname)::numeric) from pg_database;'
+ aws_upload 6710488
+ declare -r EXPECTED_SIZE=6710488
++ date +%s
+ PATH_TO_BACKUP=s3://postgres-operator-backups-bucket/spilo/test-db/aae491ea-8a1b-4d1f-b5bd-755466eae34c/logical_backups/1616113817.sql.gz
+ args=()
+ [[ ! -z 6710488 ]]
+ args+=("--expected-size=$EXPECTED_SIZE")
+ [[ ! -z '' ]]
+ [[ ! -z us-east-1 ]]
+ args+=("--region=$LOGICAL_BACKUP_S3_REGION")
+ [[ ! -z AES256 ]]
+ args+=("--sse=$LOGICAL_BACKUP_S3_SSE")
+ aws s3 cp - s3://postgres-operator-backups-bucket/spilo/test-db/aae491ea-8a1b-4d1f-b5bd-755466eae34c/logical_backups/1616113817.sql.gz --expected-size=6710488 --region=us-east-1 --sse=AES256
pg_dump: error: connection to database "template1" failed: ERROR: unexpected response from login query
pg_dumpall: error: pg_dump failed on database "template1", exiting
cluster manifest:
apiVersion: "acid.zalan.do/v1"
kind: "postgresql"
metadata:
name: "test-db"
namespace: "postgres-test"
labels:
team: test
spec:
teamId: "test"
postgresql:
version: "13"
parameters:
max_connections: "100"
numberOfInstances: 2
enableConnectionPooler: true
enableReplicaConnectionPooler: true
enableLogicalBackup: true
volume:
size: "5Gi"
users:
admin:
- superuser
- createdb
test-user: []
databases:
test: test-user
resources:
requests:
cpu: 100m
memory: 500Mi
limits:
cpu: 1000m
memory: 2000Mi
sidecars:
- name: postgres-exporter
image: wrouesnel/postgres_exporter
env:
- name: DATA_SOURCE_USER
valueFrom:
secretKeyRef:
name: postgres.test-db.credentials.postgresql.acid.zalan.do
key: username
- name: DATA_SOURCE_PASS
valueFrom:
secretKeyRef:
key: password
name: postgres.test-db.credentials.postgresql.acid.zalan.do
- name: "DATA_SOURCE_NAME"
value: "postgresql://$(DATA_SOURCE_USER):$(DATA_SOURCE_PASS)@$(POD_NAME):5432/postgres"
ports:
- containerPort: 9187
protocol: TCP
resources:
limits:
cpu: 500m
memory: 100Mi
requests:
cpu: 100m
memory: 100Mi
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the logical-backup job's pg_dumpall invocation and the manifest settings enableConnectionPooler, enableReplicaConnectionPooler, and enableLogicalBackup; reproduce the template1 failure with the supplied Kubernetes configuration. Done means logical backups complete successfully when poolers are enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, postgresql
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100