zalando / zalando/postgres-operator

Install fails in OC version 4.8 with permission issue

Open
#2,133 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? . registry.opensource.zalan.do/acid/spilo-14:2.1-p7
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? OC version 4.8.0
  • Are you running Postgres Operator in production? NO
  • Type of issue? [Bug report]

Some general remarks when posting a bug report:
My OC version

oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0     True        False         8m37s   Cluster version is 4.8.0

My config

apiVersion: v1
items:
- apiVersion: acid.zalan.do/v1
  kind: postgresql
  metadata:
    generation: 
    name: db-ds
    namespace: postgres
  spec:
    databases:
      test: testdb
    dockerImage: registry.opensource.zalan.do/acid/spilo-14:2.1-p7
    numberOfInstances: 1
    postgresql:
      version: "12"
    spiloFSGroup: 103
    spiloRunAsGroup: 103
    spiloRunAsUser: 101
    teamId: test
    users:
      dbuser:
      - superuser
    volume:
      size: 10Gi
  status:
    PostgresClusterStatus: UpdateFailed
kind: List
metadata:
  resourceVersion: ""

I'm seeing permission denied error

2022-12-07 10:12:43,419 ERROR: failed to update leader lock
2022-12-07 10:12:43,419 INFO: not promoting because failed to update leader lock in DCS
2022-12-07 10:12:49.038 32 LOG {ticks: 0, maint: 0, retry: 0}
2022-12-07 10:12:53,370 INFO: Lock owner: tsb-ds-0; I am tsb-ds-0
2022-12-07 10:12:53,419 ERROR: Permission denied
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 974, in _update_leader_with_retry
    return self._patch_or_create(self.leader_path, annotations, resource_version, ips=ips, retry=_retry)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 921, in _patch_or_create
    ret = retry(func, self._namespace, body) if retry else func(self._namespace, body)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 971, in _retry
    return retry(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/patroni/utils.py", line 334, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 483, in wrapper
    return getattr(self._core_v1_api, func)(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 419, in wrapper
    return self._api_client.call_api(method, path, headers, body, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 388, in call_api
    return self._handle_server_response(response, _preload_content)
  File "/usr/local/lib/python3.6/dist-packages/patroni/dcs/kubernetes.py", line 218, in _handle_server_response
    raise k8s_client.rest.ApiException(http_resp=response)
patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '9b001521-bf55-4d0c-a266-8d3c7ebb4fe5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'f7b3b62e-bbd5-4f55-ba41-d2bce7edfeef', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e026dd8d-e274-45b4-a104-2dc5eb0eb290', 'Date': 'Wed, 07 Dec 2022 10:12:53 GMT', 'Content-Length': '239'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"endpoints \\"tsb-ds\\" is forbidden: endpoint address 10.131.0.37 is not allowed","reason":"Forbidden","details":{"name":"tsb-ds","kind":"endpoints"},"code":403}\n'

2022-12-07 10:12:53,419 ERROR: failed to update leader lock
2022-12-07 10:12:53,419 INFO: not promoting because failed to update leader lock in DCS

I also have this clusterrole created

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: postgres-operator
    app.kubernetes.io/name: postgres-operator
  name: postgres-pod
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
- apiGroups:
  - extensions
  resourceNames:
  - privileged
  resources:
  - podsecuritypolicies
  verbs:
  - use
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
  - patch
  - get
  - list
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
  - patch
  - get
  - list
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported traceback in patroni/dcs/kubernetes.py and inspect the supplied postgres-pod ClusterRole alongside the OpenShift 4.8 403 response. Reproduce the installation using the listed image and configuration, then establish whether the endpoint permissions and reported address restriction are handled by the operator setup. Done means a verified fix or documented supported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.