zalando / zalando/postgres-operator

Logical Backup - aws CLI fails if there are no old backups to delete

Open
#2,522 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

  • Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.10.1
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
  • Are you running Postgres Operator in production? yes
  • Type of issue? Bug report

I've been trying to configure logical backups into DigitalOcean Spaces. Creating and uploading the backups work, but when I set the logical_backup_s3_retention_time config in Operator CRD, the Job that is created can fail with following logs:

+ aws_delete_outdated                                                                                                                       
+ [[ -z 7 days ]]                                                                                                                           
++ date -d '7 days ago' +%F                                                                                                                 
+ cutoff_date=2024-01-17                                                                                                                    
+ prefix=spilo/<REDACTED>/logical_backups/                                                            
+ args=("--no-paginate" "--output=text" "--prefix=$prefix" "--bucket=$LOGICAL_BACKUP_S3_BUCKET")                                            
+ [[ ! -z <REDACTED> ]]                                                                            
+ args+=("--endpoint-url=$LOGICAL_BACKUP_S3_ENDPOINT")                                                                                      
+ [[ ! -z <REDACTED> ]]                                                                                                                      
+ args+=("--region=$LOGICAL_BACKUP_S3_REGION")                                                                                              
+ aws s3api list-objects --no-paginate --output=text --prefix=spilo/<REDACTED>/logical_backups/ --buck
et=postgres-backups --endpoint-url=<REDACTED> --region=<REDACTED> '--query=Contents[?LastModified<='
\''2024-01-17'\''].[Key]'                                                                                                                   
/usr/local/lib/python3.6/dist-packages/OpenSSL/_util.py:6: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python c
ore team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.     
  from cryptography.hazmat.bindings.openssl.binding import Binding                                                                          
                                                                                                                                            
argument of type 'NoneType' is not iterable

The Job is then in Error state, and a new one is automatically created (up to 6 times which seems to be the default of Job.backoffLimit), which creates several unnecessary backups.

I presume it's because aws can't find any files that satisfy the filter, and can't iterate through the result and return its [Key].

For now, I used a workaround and created a lifecycle policy on the bucket that should work, but just wanted to report a bug.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the logical backup Job and its aws_delete_outdated cleanup path, reproducing the case where the AWS CLI returns no matching old backups. Verify the cleanup succeeds when there is nothing to delete, and confirm the Job does not enter Error or create unnecessary retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, bash, kubernetes, postgresql
Domain
cloud, databases, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.