zalando / zalando/postgres-operator
High List API Requests in a S3 Bucket
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
I have enabled the postgres-backup to the s3 bucket and with the frequency of the cronjob to run on every minute, which is obviously not a best way to implement this, But after this i faced some issues and that incur a huge cost as well.
-
My DB Size is 110 mb, but my pvc is around 30GB, which is also got completely filled up, complete 100% usage of that pvc, not sure why that happened??, but after checking got to know that , continuous backup will filled up the pg_wal directory files , not sure it is true or not ??
-
My S3 API requests are also got increased too much, see the screenshot below., not sure why that is got happened, as size in my s3 bucket will increase that is totally acceptable, but why list api operation is more rather than the put operations which is the actual thing needs to get increased when continuously backup is running.
- Operator-Image:- registry.opensource.zalan.do/acid/postgres-operator:v1.9.0
- Environment:- Kubernetes
- Type:- question
- Backup Image:- registry.opensource.zalan.do/acid/logical-backup:v1.9.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is identified. Start with the postgres-backup/logical-backup v1.9.0 setup, the one-minute cronjob, and the pg_wal directory, then inspect how its S3 interactions produce list requests. Done means explaining the PVC growth and list-versus-put behavior and identifying any required configuration or code change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, postgresql
- Domain
- cloud, databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100