zalando / zalando/postgres-operator

Allow cloning with s3 credentials from secret

Open
#2,363 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

  • Which image of the operator are you using?: registry.opensource.zalan.do/acid/postgres-operator:v1.10.0
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? OKD (OpenShift) on bare metal UPI.
  • Are you running Postgres Operator in production? yes
  • Type of issue? feature request/bug report

I have a custom s3 endpoint and s3 creds that I have in secrets.

Unfortunately only the following spec works correctly. Note that the credentials are in plain text and the configuration is duplicated. The spec below (without it) does not find any backups (because it cannot access the bucket).

kind: postgresql
metadata:
  annotations:
  labels:
    team: team
  name: team-recovery
spec:
  clone:
    cluster: team-production
    uid: ...
    timestamp: 2023-06-21T11:30:00+02:00
    s3_endpoint: https://s3.example.com
    s3_wal_path: s3://a-bucket-id/spilo/db-team-production/wal/14
    s3_force_path_style: true
    s3_access_key_id: ...
    s3_secret_access_key: ...
  env:
  - name: USE_WALG_BACKUP
    value: "true"
  - name: USE_WALG_RESTORE
    value: "true"
  - name: WAL_S3_BUCKET
    value: a-bucket-id
  - name: WALG_S3_BUCKET
    value: a-bucket-id
  - name: AWS_ENDPOINT
    value: https://s3.example.com
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        key: AWS_SECRET_ACCESS_KEY
        name: team-production-wal
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        key: AWS_ACCESS_KEY_ID
        name: team-production-wal
  numberOfInstances: 1
kind: postgresql
metadata:
  annotations:
  labels:
    team: team
  name: team-recovery
spec:
  clone:
    cluster: team-production
    uid: ...
    timestamp: 2023-06-21T11:30:00+02:00
    s3_endpoint: https://s3.example.com
    s3_wal_path: s3://a-bucket-id/spilo/db-team-production/wal/14
  env:
  - name: USE_WALG_BACKUP
    value: "true"
  - name: USE_WALG_RESTORE
    value: "true"
  - name: WAL_S3_BUCKET
    value: a-bucket-id
  - name: WALG_S3_BUCKET
    value: a-bucket-id
  - name: AWS_ENDPOINT
    value: https://s3.example.com
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        key: AWS_SECRET_ACCESS_KEY
        name: team-production-wal
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        key: AWS_ACCESS_KEY_ID
        name: team-production-wal
  numberOfInstances: 1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the postgresql spec's clone fields and the env entries using valueFrom secretKeyRef, then trace how clone configuration reaches the backup and restore process. Done means cloning can use the referenced S3 credentials without duplicating them as plaintext clone fields, including the custom endpoint and bucket shown in the examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.