zalando / zalando/postgres-operator
Problems setting up backups to s3
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
Please, answer some short questions which should help us to understand your problem / question better?
- **Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.8.1
- **Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
- **Are you running Postgres Operator in production? no
- **Type of issue? question
I am attempting to set up backups/wal archiving to an AWS s3 bucket but I cannot get it to work.
The current problem is that the wal-e env directlry does not seem to be getting set up:
envdir "/run/etc/wal-e.d/env" wal-g backup-list
chpst: fatal: unable to switch to directory: /run/etc/wal-e.d/env: file does not exist
I have been adding config values one at a time starting from the ones listed in the docs here:
Using AWS S3 or compliant services
Just adding those ones does not seem to do anything.
I have used a configmap and also set values in the helm chart:
apiVersion: v1
kind: ConfigMap
metadata:
name: pod-env-overrides
namespace: postgres-operator
data:
# Any env variable used by spilo can be added
AWS_REGION: eu-west-1
kube_iam_role: pg-backup-role
wal_s3_bucket: pg-backup-bucket
AWS_ACCESS_KEY_ID: [VALUE]
AWS_SECRET_ACCESS_KEY: [VALUE]
additional_secret_mount: aws-bucket-key
additional_secret_mount_path: /var/secrets/aws
BACKUP_NUM_TO_RETAIN: "30"
BACKUP_SCHEDULE: 0 02 * * *
USE_WALG_BACKUP: "false"
USE_WALG_RESTORE: "false"
WAL_BUCKET_SCOPE_PREFIX: ""
AWS_S3_FORCE_PATH_STYLE: "true"
WALG_DISABLE_S3_SSE: "true"
# STANDBY
STANDBY_additional_secret_mount: aws-bucket-key
STANDBY_additional_secret_mount_path: /var/secrets/aws
STANDBY_AWS_REGION: eu-west-1
STANDBY_AWS_ACCESS_KEY_ID: [VALUE]
STANDBY_AWS_SECRET_ACCESS_KEY: [VALUE]
STANDBY_WAL_S3_BUCKET: pg-backup-bucket
And the relevant section in the helm release yaml:
configAwsOrGcp:
AWS_REGION: eu-west-1
kube_iam_role: pg-backup-role
wal_s3_bucket: pg-backup-bucket
additional_secret_mount: aws-bucket-key
additional_secret_mount_path: /var/secrets/aws
STANDBY_AWS_REGION: eu-west-1
STANDBY_AWS_ACCESS_KEY_ID: [VALUE]
STANDBY_AWS_SECRET_ACCESS_KEY: [VALUE]
It seems to be populating the env variables on the pods but not setting up that env directory or trying to run wal-e/g
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the “Using AWS S3 or compliant services” administrator documentation and compare its settings with the pod-env-overrides ConfigMap and Helm configAwsOrGcp values shown here. Reproduce the envdir "/run/etc/wal-e.d/env" wal-g backup-list failure and trace why the directory is absent; done means identifying the required configuration or documenting that the setup is unsupported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, postgresql
- Domain
- cloud, database, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100