zalando / zalando/postgres-operator
Logical backups cannot use virtual-hosted S3 addressing (no way to set addressing_style)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
- Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.15.0
- Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
- Are you running Postgres Operator in production? yes
- Type of issue? feature request
Problem
logical-backup/dump.sh invokes the AWS CLI with only --endpoint-url and --region:
aws s3 cp - "$PATH_TO_BACKUP" "${args[@]//\'/}"
With --endpoint-url, botocore defaults to path-style addressing. Some S3-compatible
providers support only virtual-hosted addressing, so logical backups cannot be used
with them at all.
WAL-G backups do not have this problem: Spilo lets you pick the style through the
WALE_S3_ENDPOINT scheme (https+path://… / https+virtual://…) and derives
AWS_S3_FORCE_PATH_STYLE from it. There is no equivalent for logical backups.
Why this cannot be solved with the existing knobs
1. botocore takes addressing_style only from a config file. The environment
variable is ignored — verified on botocore 1.35.86 (image v1.14.0) and 1.42.12
(image v1.15.1). Only this works:
[default]
s3 =
addressing_style = virtual
2. The operator can inject env vars into the CronJob, but not a file.
getCronjobEnvironmentSecretVariables() produces EnvVar entries with SecretKeyRef;
there is no file-based equivalent.
3. Per-cluster volumes are not available for the backup pod.
generateLogicalBackupJob() calls generatePodTemplate() with an empty
[]acidv1.AdditionalVolume{}, so spec.additionalVolumes never reaches it.
4. additionalSecretMount does reach the pod — it is passed through from
OpConfig — so mounting a config file is technically possible. But it is a single
global operator setting, and the Secret must exist in every namespace that has
clusters. With clusters spread over dozens of namespaces this is fragile: a newly
created namespace silently breaks backups there.
Extra hazard: silent fallback
Even when addressing_style = virtual is set, botocore applies it only if the bucket
name is a single DNS label. If the name contains a dot, it silently falls back to
path-style over HTTPS — no error, no log line
(aws_is_virtual_hostable_s3_bucket, allowSubDomains=false for https).
Measured with botocore 1.35.86, addressing_style = virtual in the config file:
| bucket | resulting request URL |
|---|---|
spilo-operator.example.k8s |
https://s3.example.one/spilo-operator.example.k8s?list-type=2… (path!) |
spilo-operator-example-k8s |
https://spilo-operator-example-k8s.s3.example.one/?list-type=2… (virtual) |
For a provider that requires virtual-hosted addressing this means the request quietly
goes to the base endpoint instead of the bucket, which in our case is a different network
route entirely.
Proposed fix
A new operator option, e.g. logical_backup_s3_addressing_style (path | virtual,
empty = current behaviour), passed to the CronJob as
LOGICAL_BACKUP_S3_ADDRESSING_STYLE, plus a few lines in dump.sh:
if [[ -n "${LOGICAL_BACKUP_S3_ADDRESSING_STYLE}" ]]; then
export AWS_CONFIG_FILE=/tmp/aws-config
printf '[default]\ns3 =\n addressing_style = %s\n' \
"${LOGICAL_BACKUP_S3_ADDRESSING_STYLE}" > "${AWS_CONFIG_FILE}"
fi
Optionally, fail fast when virtual is requested for a bucket name containing a dot,
since botocore will silently ignore the setting.
We currently carry this as a wrapper around /dump.sh in a derived image. Happy to
submit a PR if the approach looks acceptable.
Related
- #3182 — default request checksums break multipart uploads on some S3-compatible
storages. Different root cause, same image; we work around it in the same wrapper with
AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED. Worth mentioning only because both
are about AWS CLI defaults that cannot be adjusted from the operator.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with logical-backup/dump.sh, getCronjobEnvironmentSecretVariables(), and generateLogicalBackupJob(); trace how OpConfig reaches the logical-backup CronJob. Confirm the option is passed into the job and that dump.sh applies the requested S3 addressing style while preserving current behavior when unset. Verify the virtual-style bucket-name hazard is handled as agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes, postgresql, shell
- Domain
- cloud, databases, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100