zalando / zalando/postgres-operator

Is it possible to assign the pv that was in use before the reboot after the reboot?

Open
#2,379 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.10.0

registry.opensource.zalan.do/acid/postgres-operator:v1.8.2

  • Where do you run it - cloud or metal? Kubernetes or OpenShift? [AWS K8s | GCP ... | Bare Metal K8s]

Bare Metal K8s ( kubeadm 1.22.3 )

  • Are you running Postgres Operator in production? [yes | no]

no ( prototyping for product )

  • Type of issue? [Bug report, question, feature request, etc.]

question

Some general remarks when posting a bug report:

  • Please, check the operator, pod (Patroni) and postgresql logs first. When copy-pasting many log lines please do it in a separate GitHub gist together with your Postgres CRD and configuration manifest.
  • If you feel this issue might be more related to the Spilo docker image or Patroni, consider opening issues in the respective repos.
Situation

I deployed postgresql cluster along with installation and it works successfully.

$ kubectl get postgresql
NAME                   TEAM   VERSION   PODS   VOLUME   CPU-REQUEST   MEMORY-REQUEST   AGE     STATUS
acid-minimal-cluster   acid   14        1      1Gi      10m           100Mi            8m23s   Running

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS      AGE
acid-minimal-cluster-0              1/1     Running   0             8m53s
postgres-operator-d59ff49cb-7swcn   1/1     Running   1 (15m ago)   15m

$ kubectl get pv
NAME                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                   STORAGECLASS    REASON   AGE
pgdata-acid-my-cluster-1   10Gi       RWO            Retain           Bound    default/pgdata-acid-minimal-cluster-0   local-storage            16m

However, if the cluster is rebooted in this state, the cluster's pod will not become ready.

$ kubectl get postgresql
NAME                   TEAM   VERSION   PODS   VOLUME   CPU-REQUEST   MEMORY-REQUEST   AGE   STATUS
acid-minimal-cluster   acid   14        1      1Gi      10m           100Mi            12m   Running
$ kubectl get pods
NAME                                READY   STATUS    RESTARTS        AGE
acid-minimal-cluster-0              0/1     Running   1 (2m32s ago)   12m
postgres-operator-d59ff49cb-7swcn   1/1     Running   2 (2m40s ago)   19m
$ kubectl get pv
NAME                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                   STORAGECLASS    REASON   AGE
pgdata-acid-my-cluster-1   10Gi       RWO            Retain           Bound    default/pgdata-acid-minimal-cluster-0   local-storage            20m

I checked the kubectl logs and it showed "waiting for leader to bootstrap" and this log continued.

2023-07-20 02:06:50,048 INFO: Lock owner: None; I am acid-minimal-cluster-0
2023-07-20 02:06:50,048 INFO: waiting for leader to bootstrap
2023-07-20 02:07:00,564 INFO: Lock owner: None; I am acid-minimal-cluster-0
2023-07-20 02:07:00,564 INFO: waiting for leader to bootstrap
entire kubectl logs result 
$ kubectl logs acid-minimal-cluster-0 

2023-07-20 02:06:46,877 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2023-07-20 02:06:46,878 - bootstrapping - INFO - Could not connect to 169.254.169.254, assuming local Docker setup
2023-07-20 02:06:46,878 - bootstrapping - INFO - No meta-data available for this provider
2023-07-20 02:06:46,879 - bootstrapping - INFO - Looks like you are running local
/scripts/configure_spilo.py:1056: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(pystache_render(TEMPLATE, placeholders))
/scripts/configure_spilo.py:1059: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  user_config = yaml.load(os.environ.get('SPILO_CONFIGURATION', os.environ.get('PATRONI_CONFIGURATION', ''))) or {}
2023-07-20 02:06:46,915 - bootstrapping - INFO - Configuring wal-e
2023-07-20 02:06:46,915 - bootstrapping - INFO - Configuring log
2023-07-20 02:06:46,915 - bootstrapping - INFO - Configuring standby-cluster
2023-07-20 02:06:46,915 - bootstrapping - INFO - Configuring bootstrap
2023-07-20 02:06:46,915 - bootstrapping - INFO - Configuring pgbouncer
2023-07-20 02:06:46,915 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2023-07-20 02:06:46,915 - bootstrapping - INFO - Configuring pgqd
2023-07-20 02:06:46,916 - bootstrapping - INFO - Configuring crontab
2023-07-20 02:06:46,916 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
/scripts/configure_spilo.py:1000: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  lines += yaml.load(placeholders['CRONTAB'])
2023-07-20 02:06:46,916 - bootstrapping - INFO - Configuring certificate
2023-07-20 02:06:46,916 - bootstrapping - INFO - Generating ssl self-signed certificate
2023-07-20 02:06:47,574 - bootstrapping - INFO - Configuring patroni
2023-07-20 02:06:47,581 - bootstrapping - INFO - Writing to file /run/postgres.yml
2023-07-20 02:06:47,581 - bootstrapping - INFO - Configuring pam-oauth2
2023-07-20 02:06:47,583 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2023-07-20 02:06:48,871 INFO: Selected new K8s API server endpoint https://192.168.63.121:6443
2023-07-20 02:06:48,872 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1f45e73c70>: Failed to establish a new connection: [Errno 113] No route to host')': /api/v1/namespaces/default/pods?labelSelector=application%3Dspilo%2Ccluster-name%3Dacid-minimal-cluster
2023-07-20 02:06:48,872 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1f45cc4070>: Failed to establish a new connection: [Errno 113] No route to host')': /api/v1/namespaces/default/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dacid-minimal-cluster
2023-07-20 02:06:48,872 ERROR: Request to server https://192.168.63.121:6443 failed: MaxRetryError("HTTPSConnectionPool(host='192.168.63.121', port=6443): Max retries exceeded with url: /api/v1/namespaces/default/pods?labelSelector=application%3Dspilo%2Ccluster-name%3Dacid-minimal-cluster (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1f45cc4310>: Failed to establish a new connection: [Errno 113] No route to host'))")
2023-07-20 02:06:48,873 ERROR: Request to server https://192.168.63.121:6443 failed: MaxRetryError("HTTPSConnectionPool(host='192.168.63.121', port=6443): Max retries exceeded with url: /api/v1/namespaces/default/endpoints?labelSelector=application%3Dspilo%2Ccluster-name%3Dacid-minimal-cluster (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1f45cc4040>: Failed to establish a new connection: [Errno 113] No route to host'))")
2023-07-20 02:06:49,886 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
2023-07-20 02:06:49,886 ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
2023-07-20 02:06:49,924 INFO: No PostgreSQL configuration items changed, nothing to reload.
2023-07-20 02:06:49,946 INFO: Lock owner: acid-minimal-cluster-0; I am acid-minimal-cluster-0
2023-07-20 02:06:49,992 INFO: Leader key released
2023-07-20 02:06:50,047 INFO: released leader key voluntarily as data dir empty and currently leader
2023-07-20 02:06:50,048 INFO: Lock owner: None; I am acid-minimal-cluster-0
2023-07-20 02:06:50,048 INFO: waiting for leader to bootstrap
2023-07-20 02:07:00,564 INFO: Lock owner: None; I am acid-minimal-cluster-0
2023-07-20 02:07:00,564 INFO: waiting for leader to bootstrap
2023-07-20 02:07:10,557 INFO: Lock owner: None; I am acid-minimal-cluster-0
Question
  1. Is it possible to use this pod's DB and PV after rebooting?
  2. If it is possible, how can I use this DB ?

Sorry if this is not the right place to ask this question.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the operator, pod (Patroni), and PostgreSQL logs, then inspect the PV/PVC state shown by kubectl get pv and the API-server connection errors. Determine whether the existing database data can be reused after reboot and document the supported steps, or identify the operator change needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.