zalando / zalando/postgres-operator

Clone: Postgresql server not started - Connection rejected

Open
#1,435 11 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

  • Which image of the operator are you using? e.g. registry.opensource.zalan.do/acid/postgres-operator:v1.6.1
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? GCP
  • Are you running Postgres Operator in production? no
  • Type of issue? Bug report

I installed postgres-operator with helm and configure pods to backup with WAL-G on a s3 minio bucket (hosted on the same kubernetes cluster):

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-pod-config
  namespace: postgres-operator
data:
  USE_WALG_BACKUP: "true"
  USE_WALG_RESTORE: "true"
  AWS_ACCESS_KEY_ID: "xxx"
  AWS_SECRET_ACCESS_KEY: "xxx"
  WALG_S3_PREFIX: "s3://postgres-backup/mydomain"
  AWS_ENDPOINT: "https://minio.mydomain.com"
  AWS_S3_FORCE_PATH_STYLE: "true"

I manage to create a cluster with 2 instances and enableConnectionPooler: true. In order to test the backup/recovery I created a table with 2 rows on it.

I ran a backup manually and then tried to restore the cluster by cloning it:

kind: "postgresql"
apiVersion: "acid.zalan.do/v1"

metadata:
  name: "acid-test-zalando-postgres-15-restored-5"
  namespace: "default"
  labels:
    team: acid

spec:
  teamId: "acid"
  postgresql:
    version: "13"
  numberOfInstances: 2
  enableConnectionPooler: true
  volume:
    size: "10Gi"


  allowedSourceRanges:
    # IP ranges to access your cluster go here
  
  resources:
    requests:
      cpu: 100m
      memory: 100Mi
    limits:
      cpu: 500m
      memory: 500Mi

  clone:
    cluster: "acid-test-zalando-postgresql-15"
    uid: "a6cd15cd-753b-457a-998b-b687be8e718e"
    timestamp: "2021-04-01T08:17:31.837+00:00"
    s3_endpoint: https://minio.mydomain.com
    s3_access_key_id: xxx
    s3_secret_access_key: xxx
    s3_force_path_style: true
    s3_wal_path: s3://postgres-backup/mydomain

But the master is never getting ready, the UI is stuck on Waiting for master to become available.
I share the logs of postgresql pods:

2021-03-31 10:00:13,171 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2021-03-31 10:00:13,180 - bootstrapping - INFO - Looks like your running google
2021-03-31 10:00:13,262 - bootstrapping - INFO - Configuring patroni
2021-03-31 10:00:13,276 - bootstrapping - INFO - Writing to file /run/postgres.yml
2021-03-31 10:00:13,277 - bootstrapping - INFO - Configuring pgqd
2021-03-31 10:00:13,277 - bootstrapping - INFO - Configuring log
2021-03-31 10:00:13,278 - bootstrapping - INFO - Configuring standby-cluster
2021-03-31 10:00:13,278 - bootstrapping - INFO - Configuring pam-oauth2
2021-03-31 10:00:13,278 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2021-03-31 10:00:13,278 - bootstrapping - INFO - Configuring wal-e
2021-03-31 10:00:13,279 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_S3_PREFIX
2021-03-31 10:00:13,279 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_S3_PREFIX
2021-03-31 10:00:13,279 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_ACCESS_KEY_ID
2021-03-31 10:00:13,279 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
2021-03-31 10:00:13,280 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_S3_ENDPOINT
2021-03-31 10:00:13,280 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_ENDPOINT
2021-03-31 10:00:13,280 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_DISABLE_S3_SSE
2021-03-31 10:00:13,280 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/AWS_S3_FORCE_PATH_STYLE
2021-03-31 10:00:13,280 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_DOWNLOAD_CONCURRENCY
2021-03-31 10:00:13,281 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALG_UPLOAD_CONCURRENCY
2021-03-31 10:00:13,281 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/USE_WALG_BACKUP
2021-03-31 10:00:13,281 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/USE_WALG_RESTORE
2021-03-31 10:00:13,281 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/WALE_LOG_DESTINATION
2021-03-31 10:00:13,281 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/BACKUP_NUM_TO_RETAIN
2021-03-31 10:00:13,282 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env/TMPDIR
2021-03-31 10:00:13,283 - bootstrapping - INFO - Configuring certificate
2021-03-31 10:00:13,283 - bootstrapping - INFO - Generating ssl certificate
2021-03-31 10:00:13,478 - bootstrapping - INFO - Configuring crontab
2021-03-31 10:00:13,496 - bootstrapping - INFO - Configuring pgbouncer
2021-03-31 10:00:13,496 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2021-03-31 10:00:13,496 - bootstrapping - INFO - Configuring bootstrap
2021-03-31 10:00:13,497 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/WALE_S3_PREFIX
2021-03-31 10:00:13,497 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/WALG_S3_PREFIX
2021-03-31 10:00:13,497 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/AWS_ACCESS_KEY_ID
2021-03-31 10:00:13,497 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/AWS_SECRET_ACCESS_KEY
2021-03-31 10:00:13,497 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/WALE_S3_ENDPOINT
2021-03-31 10:00:13,498 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/AWS_ENDPOINT
2021-03-31 10:00:13,498 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/WALG_DISABLE_S3_SSE
2021-03-31 10:00:13,498 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/AWS_S3_FORCE_PATH_STYLE
2021-03-31 10:00:13,499 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/USE_WALG_BACKUP
2021-03-31 10:00:13,499 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/USE_WALG_RESTORE
2021-03-31 10:00:13,499 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/WALE_LOG_DESTINATION
2021-03-31 10:00:13,499 - bootstrapping - INFO - Writing to file /run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15/TMPDIR
2021-03-31 10:00:15,464 INFO: Selected new K8s API server endpoint https://35.205.59.174:443
2021-03-31 10:00:15,485 INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-03-31 10:00:15,488 INFO: Lock owner: None; I am acid-test-zalando-postgres-15-restored-5-0
2021-03-31 10:00:15,600 INFO: trying to bootstrap a new cluster
2021-03-31 10:00:15,600 INFO: Running custom bootstrap script: envdir "/run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15" python3 /scripts/clone_with_wale.py --recovery-target-time="2021-04-01T08:17:31.837+00:00"
2021-03-31 10:00:15,667 INFO: Trying s3://postgres-backup/adisoft for clone
2021-03-31 10:00:15,875 INFO: cloning cluster acid-test-zalando-postgres-15-restored-5 using wal-g backup-fetch /home/postgres/pgdata/pgroot/data base_000000010000000000000002
INFO: 2021/03/31 10:00:16.164564 Finished decompression of part_003.tar.lz4
INFO: 2021/03/31 10:00:16.164585 Finished extraction of part_003.tar.lz4
INFO: 2021/03/31 10:00:18.935429 Finished decompression of part_001.tar.lz4
INFO: 2021/03/31 10:00:18.935450 Finished extraction of part_001.tar.lz4
INFO: 2021/03/31 10:00:18.947468 Finished decompression of pg_control.tar.lz4
INFO: 2021/03/31 10:00:18.947511 Finished extraction of pg_control.tar.lz4
INFO: 2021/03/31 10:00:18.947534 
Backup extraction complete.
2021-03-31 10:00:19,463 maybe_pg_upgrade INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-03-31 10:00:20 UTC [107]: [1-1] 60644834.6b 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2021-03-31 10:00:20 UTC [107]: [2-1] 60644834.6b 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2021-03-31 10:00:20,347 INFO: postmaster pid=107
2021-03-31 10:00:20 UTC [107]: [3-1] 60644834.6b 0     LOG:  redirecting log output to logging collector process
2021-03-31 10:00:20 UTC [107]: [4-1] 60644834.6b 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - no response
2021-03-31 10:00:24,794 INFO: removing initialize key after failed attempt to bootstrap the cluster
2021-03-31 10:00:24,843 INFO: renaming data directory to /home/postgres/pgdata/pgroot/data_2021-03-31-10-00-24
Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 33, in <module>
    sys.exit(load_entry_point('patroni==2.0.1', 'console_scripts', 'patroni')())
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 170, in main
    return patroni_main()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 138, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 100, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 108, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 59, in run
    self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 111, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1457, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1351, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1247, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1240, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
/run/service/patroni: finished with code=1 signal=0
/run/service/patroni: sleeping 30 seconds
2021-03-31 10:00:55,893 INFO: Selected new K8s API server endpoint https://35.205.59.174:443
2021-03-31 10:00:55,949 INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-03-31 10:00:55,954 INFO: Lock owner: None; I am acid-test-zalando-postgres-15-restored-5-0
2021-03-31 10:00:56,128 INFO: trying to bootstrap a new cluster
2021-03-31 10:00:56,128 INFO: Running custom bootstrap script: envdir "/run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15" python3 /scripts/clone_with_wale.py --recovery-target-time="2021-04-01T08:17:31.837+00:00"
2021-03-31 10:00:56,196 INFO: Trying s3://postgres-backup/adisoft for clone
2021-03-31 10:00:56,451 INFO: cloning cluster acid-test-zalando-postgres-15-restored-5 using wal-g backup-fetch /home/postgres/pgdata/pgroot/data base_000000010000000000000002
INFO: 2021/03/31 10:00:56.759242 Finished decompression of part_003.tar.lz4
INFO: 2021/03/31 10:00:56.759276 Finished extraction of part_003.tar.lz4
INFO: 2021/03/31 10:00:59.305742 Finished decompression of part_001.tar.lz4
INFO: 2021/03/31 10:00:59.305810 Finished extraction of part_001.tar.lz4
INFO: 2021/03/31 10:00:59.314427 Finished decompression of pg_control.tar.lz4
INFO: 2021/03/31 10:00:59.314523 Finished extraction of pg_control.tar.lz4
INFO: 2021/03/31 10:00:59.314535 
Backup extraction complete.
2021-03-31 10:00:59,759 maybe_pg_upgrade INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-03-31 10:01:00 UTC [284]: [1-1] 6064485c.11c 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2021-03-31 10:01:00 UTC [284]: [2-1] 6064485c.11c 0     LOG:  pg_stat_kcache.linux_hz is set to 333333
2021-03-31 10:01:00,664 INFO: postmaster pid=284
2021-03-31 10:01:00 UTC [284]: [3-1] 6064485c.11c 0     LOG:  redirecting log output to logging collector process
2021-03-31 10:01:00 UTC [284]: [4-1] 6064485c.11c 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - no response
2021-03-31 10:01:05,488 INFO: removing initialize key after failed attempt to bootstrap the cluster
2021-03-31 10:01:05,536 INFO: renaming data directory to /home/postgres/pgdata/pgroot/data_2021-03-31-10-01-05
Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 33, in <module>
    sys.exit(load_entry_point('patroni==2.0.1', 'console_scripts', 'patroni')())
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 170, in main
    return patroni_main()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 138, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 100, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 108, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 59, in run
    self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 111, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1457, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1351, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1247, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1240, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
/run/service/patroni: finished with code=1 signal=0
/run/service/patroni: sleeping 60 seconds
2021-03-31 10:02:06,948 INFO: Selected new K8s API server endpoint https://35.205.59.174:443
2021-03-31 10:02:06,964 INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-03-31 10:02:06,968 INFO: Lock owner: None; I am acid-test-zalando-postgres-15-restored-5-0
2021-03-31 10:02:07,082 INFO: trying to bootstrap a new cluster
2021-03-31 10:02:07,082 INFO: Running custom bootstrap script: envdir "/run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15" python3 /scripts/clone_with_wale.py --recovery-target-time="2021-04-01T08:17:31.837+00:00"
2021-03-31 10:02:07,154 INFO: Trying s3://postgres-backup/adisoft for clone
2021-03-31 10:02:07,374 INFO: cloning cluster acid-test-zalando-postgres-15-restored-5 using wal-g backup-fetch /home/postgres/pgdata/pgroot/data base_000000010000000000000002
INFO: 2021/03/31 10:02:07.680876 Finished decompression of part_003.tar.lz4
INFO: 2021/03/31 10:02:07.680898 Finished extraction of part_003.tar.lz4
INFO: 2021/03/31 10:02:10.306694 Finished decompression of part_001.tar.lz4
INFO: 2021/03/31 10:02:10.306713 Finished extraction of part_001.tar.lz4
INFO: 2021/03/31 10:02:10.316917 Finished decompression of pg_control.tar.lz4
INFO: 2021/03/31 10:02:10.316933 Finished extraction of pg_control.tar.lz4
INFO: 2021/03/31 10:02:10.316942 
Backup extraction complete.
2021-03-31 10:02:10,779 maybe_pg_upgrade INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-03-31 10:02:11 UTC [468]: [1-1] 606448a3.1d4 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2021-03-31 10:02:11 UTC [468]: [2-1] 606448a3.1d4 0     LOG:  pg_stat_kcache.linux_hz is set to 500000
2021-03-31 10:02:11,563 INFO: postmaster pid=468
2021-03-31 10:02:11 UTC [468]: [3-1] 606448a3.1d4 0     LOG:  redirecting log output to logging collector process
2021-03-31 10:02:11 UTC [468]: [4-1] 606448a3.1d4 0     HINT:  Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - no response
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - no response
2021-03-31 10:02:16,370 INFO: removing initialize key after failed attempt to bootstrap the cluster
2021-03-31 10:02:16,425 INFO: renaming data directory to /home/postgres/pgdata/pgroot/data_2021-03-31-10-02-16
Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 33, in <module>
    sys.exit(load_entry_point('patroni==2.0.1', 'console_scripts', 'patroni')())
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 170, in main
    return patroni_main()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 138, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 100, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 108, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 59, in run
    self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 111, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1457, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1351, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1247, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1240, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
/run/service/patroni: finished with code=1 signal=0
/run/service/patroni: sleeping 90 seconds
2021-03-31 10:03:47,385 INFO: Selected new K8s API server endpoint https://35.205.59.174:443
2021-03-31 10:03:47,406 INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-03-31 10:03:47,412 INFO: Lock owner: None; I am acid-test-zalando-postgres-15-restored-5-0
2021-03-31 10:03:47,559 INFO: trying to bootstrap a new cluster
2021-03-31 10:03:47,560 INFO: Running custom bootstrap script: envdir "/run/etc/wal-e.d/env-clone-acid-test-zalando-postgresql-15" python3 /scripts/clone_with_wale.py --recovery-target-time="2021-04-01T08:17:31.837+00:00"
2021-03-31 10:03:47,670 INFO: Trying s3://postgres-backup/adisoft for clone
2021-03-31 10:03:47,878 INFO: cloning cluster acid-test-zalando-postgres-15-restored-5 using wal-g backup-fetch /home/postgres/pgdata/pgroot/data base_000000010000000000000002
INFO: 2021/03/31 10:03:48.250101 Finished decompression of part_003.tar.lz4
INFO: 2021/03/31 10:03:48.250123 Finished extraction of part_003.tar.lz4
INFO: 2021/03/31 10:03:50.664832 Finished decompression of part_001.tar.lz4
INFO: 2021/03/31 10:03:50.664853 Finished extraction of part_001.tar.lz4
INFO: 2021/03/31 10:03:50.674144 Finished decompression of pg_control.tar.lz4
INFO: 2021/03/31 10:03:50.674161 Finished extraction of pg_control.tar.lz4
INFO: 2021/03/31 10:03:50.674170 
Backup extraction complete.
2021-03-31 10:03:51,197 maybe_pg_upgrade INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-03-31 10:03:52 UTC [662]: [1-1] 60644908.296 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
2021-03-31 10:03:52 UTC [662]: [2-1] 60644908.296 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2021-03-31 10:03:52 UTC [662]: [3-1] 60644908.296 0     LOG:  redirecting log output to logging collector process
2021-03-31 10:03:52 UTC [662]: [4-1] 60644908.296 0     HINT:  Future log output will appear in directory "../pg_log".
2021-03-31 10:03:52,248 INFO: postmaster pid=662
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - no response
2021-03-31 10:03:56,583 INFO: removing initialize key after failed attempt to bootstrap the cluster
2021-03-31 10:03:56,632 INFO: renaming data directory to /home/postgres/pgdata/pgroot/data_2021-03-31-10-03-56
Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 33, in <module>
    sys.exit(load_entry_point('patroni==2.0.1', 'console_scripts', 'patroni')())
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 170, in main
    return patroni_main()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 138, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 100, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 108, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 59, in run
    self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 111, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1457, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1351, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1247, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1240, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
/run/service/patroni: finished with code=1 signal=0
/run/service/patroni: sleeping 120 seconds

I exec in one of the postgresql pod and try to connect to postgresql server by using psql but I'm facing

psql: error: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Any idea what's wrong here ?

Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with /scripts/clone_with_wale.py and the Patroni bootstrap output for the clone path, then compare the WAL-G environment and recovery-target details in the report. Done means the cloned PostgreSQL master completes bootstrap, accepts connections, and the restored table is available.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.