zalando / zalando/postgres-operator

High List API Requests in a S3 Bucket

Open
#2,631 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

I have enabled the postgres-backup to the s3 bucket and with the frequency of the cronjob to run on every minute, which is obviously not a best way to implement this, But after this i faced some issues and that incur a huge cost as well.

  1. My DB Size is 110 mb, but my pvc is around 30GB, which is also got completely filled up, complete 100% usage of that pvc, not sure why that happened??, but after checking got to know that , continuous backup will filled up the pg_wal directory files , not sure it is true or not ??

  2. My S3 API requests are also got increased too much, see the screenshot below., not sure why that is got happened, as size in my s3 bucket will increase that is totally acceptable, but why list api operation is more rather than the put operations which is the actual thing needs to get increased when continuously backup is running.

Screenshot 2024-05-19 at 16 37 11 Screenshot 2024-05-19 at 16 37 34
  • Operator-Image:- registry.opensource.zalan.do/acid/postgres-operator:v1.9.0
  • Environment:- Kubernetes
  • Type:- question
  • Backup Image:- registry.opensource.zalan.do/acid/logical-backup:v1.9.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is identified. Start with the postgres-backup/logical-backup v1.9.0 setup, the one-minute cronjob, and the pg_wal directory, then inspect how its S3 interactions produce list requests. Done means explaining the PVC growth and list-versus-put behavior and identifying any required configuration or code change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, postgresql
Domain
cloud, databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.