zalando / zalando/postgres-operator
`AWS_REQUEST_CHECKSUM_CALCULATION` to fix Ceph/S3 multipart upload crashes
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
I started having issues with logical backups failing on a database of 70G after upgrading our Ceph cluster from Squid to Tentacle. The backups were crashing with a TypeError: argument of type 'NoneType' is not iterable boto3 error during the multipart upload to S3.
I did a bunch of testing that led to a working manual backup script where I pinned the AWS CLI to version 2.22.34. This successfully bypassed the newer AWS CLI behavior that forces CRC64NVME checksums on multipart uploads, which our Ceph endpoint couldn't handle.
After digging deeper into upstream AWS CLI and boto3 pull requests, I found the environment variable AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED. I tested injecting this via our pod-config secret using the official v2.0.2 logical backup image, and it works perfectly.
Since I couldn't find any reference to this variable in the documentation, is this worth adding a note for anyone using Ceph RGW or other S3-compatible providers hitting this exact multipart upload crash? If not I hope that at least this issue might help the next person who runs into this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the logical backup documentation and the pod-config secret configuration described in the issue. Document AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED for Ceph RGW or other S3-compatible providers, and consider the work done when users can find and apply the setting to avoid the multipart upload crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, postgresql
- Domain
- cloud, databases, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 74/100