zalando / zalando/postgres-operator
Logical Backup - aws CLI fails if there are no old backups to delete
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
- Which image of the operator are you using?
registry.opensource.zalan.do/acid/postgres-operator:v1.10.1 - Where do you run it - cloud or metal? Kubernetes or OpenShift?
Bare Metal K8s - Are you running Postgres Operator in production?
yes - Type of issue?
Bug report
I've been trying to configure logical backups into DigitalOcean Spaces. Creating and uploading the backups work, but when I set the logical_backup_s3_retention_time config in Operator CRD, the Job that is created can fail with following logs:
+ aws_delete_outdated
+ [[ -z 7 days ]]
++ date -d '7 days ago' +%F
+ cutoff_date=2024-01-17
+ prefix=spilo/<REDACTED>/logical_backups/
+ args=("--no-paginate" "--output=text" "--prefix=$prefix" "--bucket=$LOGICAL_BACKUP_S3_BUCKET")
+ [[ ! -z <REDACTED> ]]
+ args+=("--endpoint-url=$LOGICAL_BACKUP_S3_ENDPOINT")
+ [[ ! -z <REDACTED> ]]
+ args+=("--region=$LOGICAL_BACKUP_S3_REGION")
+ aws s3api list-objects --no-paginate --output=text --prefix=spilo/<REDACTED>/logical_backups/ --buck
et=postgres-backups --endpoint-url=<REDACTED> --region=<REDACTED> '--query=Contents[?LastModified<='
\''2024-01-17'\''].[Key]'
/usr/local/lib/python3.6/dist-packages/OpenSSL/_util.py:6: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python c
ore team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
from cryptography.hazmat.bindings.openssl.binding import Binding
argument of type 'NoneType' is not iterable
The Job is then in Error state, and a new one is automatically created (up to 6 times which seems to be the default of Job.backoffLimit), which creates several unnecessary backups.
I presume it's because aws can't find any files that satisfy the filter, and can't iterate through the result and return its [Key].
For now, I used a workaround and created a lifecycle policy on the bucket that should work, but just wanted to report a bug.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the logical backup Job and its aws_delete_outdated cleanup path, reproducing the case where the AWS CLI returns no matching old backups. Verify the cleanup succeeds when there is nothing to delete, and confirm the Job does not enter Error or create unnecessary retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, bash, kubernetes, postgresql
- Domain
- cloud, databases, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100