zalando / zalando/postgres-operator

Logical backups cannot use virtual-hosted S3 addressing (no way to set addressing_style)

Open
#3,183 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

  • Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.15.0
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
  • Are you running Postgres Operator in production? yes
  • Type of issue? feature request

Problem

logical-backup/dump.sh invokes the AWS CLI with only --endpoint-url and --region:

aws s3 cp - "$PATH_TO_BACKUP" "${args[@]//\'/}"

With --endpoint-url, botocore defaults to path-style addressing. Some S3-compatible
providers support only virtual-hosted addressing, so logical backups cannot be used
with them at all.

WAL-G backups do not have this problem: Spilo lets you pick the style through the
WALE_S3_ENDPOINT scheme (https+path://… / https+virtual://…) and derives
AWS_S3_FORCE_PATH_STYLE from it. There is no equivalent for logical backups.

Why this cannot be solved with the existing knobs

1. botocore takes addressing_style only from a config file. The environment
variable is ignored — verified on botocore 1.35.86 (image v1.14.0) and 1.42.12
(image v1.15.1). Only this works:

[default]
s3 =
    addressing_style = virtual

2. The operator can inject env vars into the CronJob, but not a file.
getCronjobEnvironmentSecretVariables() produces EnvVar entries with SecretKeyRef;
there is no file-based equivalent.

3. Per-cluster volumes are not available for the backup pod.
generateLogicalBackupJob() calls generatePodTemplate() with an empty
[]acidv1.AdditionalVolume{}, so spec.additionalVolumes never reaches it.

4. additionalSecretMount does reach the pod — it is passed through from
OpConfig — so mounting a config file is technically possible. But it is a single
global operator setting, and the Secret must exist in every namespace that has
clusters. With clusters spread over dozens of namespaces this is fragile: a newly
created namespace silently breaks backups there.

Extra hazard: silent fallback

Even when addressing_style = virtual is set, botocore applies it only if the bucket
name is a single DNS label
. If the name contains a dot, it silently falls back to
path-style over HTTPS — no error, no log line
(aws_is_virtual_hostable_s3_bucket, allowSubDomains=false for https).

Measured with botocore 1.35.86, addressing_style = virtual in the config file:

bucket resulting request URL
spilo-operator.example.k8s https://s3.example.one/spilo-operator.example.k8s?list-type=2… (path!)
spilo-operator-example-k8s https://spilo-operator-example-k8s.s3.example.one/?list-type=2… (virtual)

For a provider that requires virtual-hosted addressing this means the request quietly
goes to the base endpoint instead of the bucket, which in our case is a different network
route entirely.

Proposed fix

A new operator option, e.g. logical_backup_s3_addressing_style (path | virtual,
empty = current behaviour), passed to the CronJob as
LOGICAL_BACKUP_S3_ADDRESSING_STYLE, plus a few lines in dump.sh:

if [[ -n "${LOGICAL_BACKUP_S3_ADDRESSING_STYLE}" ]]; then
    export AWS_CONFIG_FILE=/tmp/aws-config
    printf '[default]\ns3 =\n    addressing_style = %s\n' \
        "${LOGICAL_BACKUP_S3_ADDRESSING_STYLE}" > "${AWS_CONFIG_FILE}"
fi

Optionally, fail fast when virtual is requested for a bucket name containing a dot,
since botocore will silently ignore the setting.

We currently carry this as a wrapper around /dump.sh in a derived image. Happy to
submit a PR if the approach looks acceptable.

Related

  • #3182 — default request checksums break multipart uploads on some S3-compatible
    storages. Different root cause, same image; we work around it in the same wrapper with
    AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED. Worth mentioning only because both
    are about AWS CLI defaults that cannot be adjusted from the operator.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with logical-backup/dump.sh, getCronjobEnvironmentSecretVariables(), and generateLogicalBackupJob(); trace how OpConfig reaches the logical-backup CronJob. Confirm the option is passed into the job and that dump.sh applies the requested S3 addressing style while preserving current behavior when unset. Verify the virtual-style bucket-name hazard is handled as agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes, postgresql, shell
Domain
cloud, databases, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.