zalando / zalando/postgres-operator

if you use etcd_host, one key is used for different namespaces

Open
#2,024 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.8.2
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
  • Are you running Postgres Operator in production? yes
  • Type of issue? Bug report

Some general remarks when posting a bug report:

  • Please, check the operator, pod (Patroni) and postgresql logs first. When copy-pasting many log lines please do it in a separate GitHub gist together with your Postgres CRD and configuration manifest.
  • If you feel this issue might be more related to the Spilo docker image or Patroni, consider opening issues in the respective repos.

there are two problems:

  1. the problem manifests itself with the same name and different namespace if use etcd_host
  2. deleting cr does not delete the entry in etcd about the psql cluster, which makes it impossible to create the same cluster over a new one.

operator env

  configGeneral:
    etcd_host: etcd.{{ .Release.Namespace }}.svc.{{ .Values.configKubernetes.cluster_domain }}

test psql-test-01 or psql-test-02 don't start
for psql-test-03 I found a hack solution through an additional env

  env:
    - name: NAMESPACE
      value: /psql-test-03

I think the first problem is related to this code

test.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: psql-test-01
spec:
  finalizers:
  - kubernetes
---
apiVersion: v1
kind: Namespace
metadata:
  name: psql-test-02
spec:
  finalizers:
  - kubernetes
---
apiVersion: v1
kind: Namespace
metadata:
  name: psql-test-03
spec:
  finalizers:
  - kubernetes
---
apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: test-psql
  namespace: psql-test-01
spec:
  allowedSourceRanges:
  - 0.0.0.0/0
  databases:
    catalogu: zalando
    init: zalando
  enableShmVolume: true
  numberOfInstances: 1
  patroni:
    pg_hba:
    - host      all         okagent all md5
    - local     all         all trust
    - hostssl   all         +zalandos 127.0.0.1/32 pam
    - host      all         all 127.0.0.1/32 md5
    - hostssl   all         +zalandos ::1/128 pam
    - host      all         all ::1/128 md5
    - local     replication standby trust
    - hostssl   replication standby all md5
    - hostnossl all         all all reject
    - hostssl   all         +zalandos all pam
    - hostssl   all         all all md5
  podAnnotations:
    threshold.extended-monitoring.flant.com/disk-bytes-critical: "70"
    threshold.extended-monitoring.flant.com/disk-bytes-warning: "50"
  postgresql:
    parameters:
      hot_standby_feedback: "on"
      max_standby_archive_delay: 300s
      max_standby_streaming_delay: 300s
      pg_stat_statements.max: "500"
      pg_stat_statements.save: "false"
      pg_stat_statements.track: top
      pg_stat_statements.track_utility: "true"
      track_io_timing: "on"
    version: "13"
  resources:
    limits:
      cpu: 2000m
      memory: 256Mi
    requests:
      cpu: 50m
      memory: 256Mi
  teamId: test
  users:
    foo_user: []
    zalando:
    - superuser
    - createdb
  volume:
    size: 10Gi
---
apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: test-psql
  namespace: psql-test-02
spec:
  allowedSourceRanges:
  - 0.0.0.0/0
  databases:
    catalogu: zalando
    init: zalando
  enableShmVolume: true
  numberOfInstances: 1
  patroni:
    pg_hba:
    - host      all         okagent all md5
    - local     all         all trust
    - hostssl   all         +zalandos 127.0.0.1/32 pam
    - host      all         all 127.0.0.1/32 md5
    - hostssl   all         +zalandos ::1/128 pam
    - host      all         all ::1/128 md5
    - local     replication standby trust
    - hostssl   replication standby all md5
    - hostnossl all         all all reject
    - hostssl   all         +zalandos all pam
    - hostssl   all         all all md5
  podAnnotations:
    threshold.extended-monitoring.flant.com/disk-bytes-critical: "70"
    threshold.extended-monitoring.flant.com/disk-bytes-warning: "50"
  postgresql:
    parameters:
      hot_standby_feedback: "on"
      max_standby_archive_delay: 300s
      max_standby_streaming_delay: 300s
      pg_stat_statements.max: "500"
      pg_stat_statements.save: "false"
      pg_stat_statements.track: top
      pg_stat_statements.track_utility: "true"
      track_io_timing: "on"
    version: "13"
  resources:
    limits:
      cpu: 2000m
      memory: 256Mi
    requests:
      cpu: 50m
      memory: 256Mi
  teamId: test
  users:
    foo_user: []
    zalando:
    - superuser
    - createdb
  volume:
    size: 10Gi
---
apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: test-psql
  namespace: psql-test-03
spec:
  allowedSourceRanges:
  - 0.0.0.0/0
  databases:
    catalogu: zalando
    init: zalando
  enableShmVolume: true
  numberOfInstances: 1
  env:
    - name: NAMESPACE
      value: /psql-test-03 
  patroni:
    pg_hba:
    - host      all         okagent all md5
    - local     all         all trust
    - hostssl   all         +zalandos 127.0.0.1/32 pam
    - host      all         all 127.0.0.1/32 md5
    - hostssl   all         +zalandos ::1/128 pam
    - host      all         all ::1/128 md5
    - local     replication standby trust
    - hostssl   replication standby all md5
    - hostnossl all         all all reject
    - hostssl   all         +zalandos all pam
    - hostssl   all         all all md5
  podAnnotations:
    threshold.extended-monitoring.flant.com/disk-bytes-critical: "70"
    threshold.extended-monitoring.flant.com/disk-bytes-warning: "50"
  postgresql:
    parameters:
      hot_standby_feedback: "on"
      max_standby_archive_delay: 300s
      max_standby_streaming_delay: 300s
      pg_stat_statements.max: "500"
      pg_stat_statements.save: "false"
      pg_stat_statements.track: top
      pg_stat_statements.track_utility: "true"
      track_io_timing: "on"
    version: "13"
  resources:
    limits:
      cpu: 2000m
      memory: 256Mi
    requests:
      cpu: 50m
      memory: 256Mi
  teamId: test
  users:
    foo_user: []
    zalando:
    - superuser
    - createdb
  volume:
    size: 10Gi

root@test-psql-0:/home/postgres# ETCDCTL_API=2 etcdctl --endpoints http://etcd.postgres-operator-bravo-shared.svc.cluster.local:2379 ls --recursive
/service
/service/test-psql
/service/test-psql/members
/service/test-psql/members/test-psql-0
/service/test-psql/initialize
/service/test-psql/config
/service/test-psql/leader
/service/test-psql/status
/psql-test-03
/psql-test-03/test-psql
/psql-test-03/test-psql/members
/psql-test-03/test-psql/members/test-psql-0
/psql-test-03/test-psql/initialize
/psql-test-03/test-psql/config
/psql-test-03/test-psql/leader
/psql-test-03/test-psql/status

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked configure_spilo.py lines 555-556 and the operator's handling of the etcd_host setting. Reproduce the namespace collision and deletion behavior using test.yaml, then inspect the etcdctl tree shown in the report. Done means same-named clusters can coexist across namespaces and deleting a CR removes its etcd entry.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.