zalando / zalando/postgres-operator

Add a way to to make the logical backups rsyncable

Open
#1,962 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.8.2
  • Where do you run it ? GCP
  • Are you running Postgres Operator in production? yes
  • Type of issue? feature request

We need to replicate the logical backups and it turns out to be quite costly in egress fees as the compression is not done with the --rsyncable option.

Adding the --rsyncable to pigz makes the archive much better when you need to rsync or rclone them.

This can be done by adding a PIGZ="--rsyncable" environment variable to the logical backup cronjob, but there is no way yo do this.

I also tried to make our own docker image based on registry.opensource.zalan.do/acid/logical-backup:v1.8.0, but here again, I can use the configLogicalBackup.logical_backup_docker_image parameter to use our image, but since there is no parameter for the imagePullSecrets, this doesn't work either.

So please, could you:

  1. Add the --rsyncable parameter to the pigz command in dump.sh
  2. And/Or add a logical_backup_env parameter that would allow passing parameters to the pigz
  3. And/Or add a logical_backup_image_pull_secrets parameter.

Each of those solutions would allow resolving this problem, but ideally, all 3 would give much more flexibility.

Thank you !

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dump.sh and the logical backup cronjob configuration. Trace how configLogicalBackup.logical_backup_docker_image is applied, then check how environment variables and imagePullSecrets could be exposed. Done means a supported configuration can enable rsyncable pigz compression and, where applicable, use a custom image with its pull secrets.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.