zalando / zalando/postgres-operator

Operator instantiates clusters that don't ever come online

Open
#1,608 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Which image of the operator are you using?
registry.opensource.zalan.do/acid/postgres-operator:v1.7.0 (sha256:9c3bafdd67fd64e27caba808f38112acfdc725fb6248bb0df8489e4b1020346c)

Where do you run it - cloud or metal? Kubernetes or OpenShift?
Bare Metal K8s (kubeadm)

Are you running Postgres Operator in production?
No, just testing it out atm

Type of issue?
I'm suspecting a bug since not even the minimal manifest example (from the User Guide) starts properly.

Issue Description
I've tried to use the 1.7.0 operator release on a bare metal kubeadm cluster to manage database deployments with. The operator deployment itself seems to work fine; it picks up Postgres kinds, deletes them, etc. The database clusters it's creating however, do not start at all. Even de example minimal manifest does not start up.
The minimal manifest was only edited to set the spec.volume.storageClass value so it would get the correct storage assigned. No other changes.

Operator Deployment Information
The operator was deployed using Helm (3).

helm repo add zalando-postgres https://opensource.zalando.com/postgres-operator/charts/postgres-operator/

helm install postgres-operator zalando-postgres/postgres-operator \
  --values postgres-operator-values.yaml

With the following in postgres-operator-values.yaml:

configGeneral:
  # Require SHM volume overrides to be explicit on the postgres manifests
  enable_shm_volume: false

configKubernetes:
  # Enable anti-affinity by default so HA deployments would actually be guaranteed more HA than not if a node fails
  enable_pod_antiaffinity: true

podServiceAccount:
  # Have it generate a name so multiple deployments of the operator don't overlap their service accounts
  name: null

# Suppress Helm 2 hook warning
crd:
  create: false

Operator Logs

time="2021-09-03T11:41:34Z" level=info msg="Spilo operator v1.7.0\n"
time="2021-09-03T11:41:34Z" level=error msg="could not create customResourceDefinition \"operatorconfigurations.acid.zalan.do\": CustomResourceDefinition.apiextensions.k8s.io \"operatorconfigurations.acid.zalan.do\" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required" pkg=controller
time="2021-09-03T11:41:38Z" level=warning msg="in the operator configuration, the pod service account name postgres-operator-pod does not match the name postgres-pod given in the account definition; using the former for consistency" pkg=controller
time="2021-09-03T11:41:38Z" level=info msg="Parse role bindings" pkg=controller
time="2021-09-03T11:41:38Z" level=info msg="successfully parsed" pkg=controller
time="2021-09-03T11:41:38Z" level=info msg="Listening to all namespaces" pkg=controller
time="2021-09-03T11:41:38Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered and will only be updated" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="{" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ReadyWaitInterval\": 3000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ReadyWaitTimeout\": 30000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ResyncPeriod\": 1800000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"RepairPeriod\": 300000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableCRDValidation\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ResourceCheckInterval\": 3000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ResourceCheckTimeout\": 600000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodLabelWaitTimeout\": 600000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodDeletionWaitTimeout\": 600000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SpiloRunAsUser\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SpiloRunAsGroup\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SpiloFSGroup\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodPriorityClassName\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ClusterDomain\": \"cluster.local\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SpiloPrivileged\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SpiloAllowPrivilegeEscalation\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"AdditionalPodCapabilities\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ClusterLabels\": {" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="      \"application\": \"spilo\"" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   }," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"InheritedLabels\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"InheritedAnnotations\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DownscalerAnnotations\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ClusterNameLabel\": \"cluster-name\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DeleteAnnotationDateKey\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DeleteAnnotationNameKey\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodRoleLabel\": \"spilo-role\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodToleration\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DefaultCPURequest\": \"100m\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DefaultCPULimit\": \"1\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DefaultMemoryLimit\": \"500Mi\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MinCPULimit\": \"250m\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MinMemoryLimit\": \"250Mi\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodEnvironmentConfigMap\": \"/\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodEnvironmentSecret\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"NodeReadinessLabel\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MaxInstances\": -1," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MinInstances\": -1," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ShmVolume\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SecretNameTemplate\": \"{username}.{cluster}.credentials.{tprkind}.{tprgroup}\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PamRoleName\": \"zalandos\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PamConfiguration\": \"https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"TeamsAPIUrl\": \"https://teams.example.com/api/\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"OAuthTokenSecretName\": \"storage-operators/postgres-operator\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"InfrastructureRolesSecretName\": \"/\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"InfrastructureRoles\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"InfrastructureRolesDefs\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SuperUsername\": \"postgres\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ReplicationUsername\": \"standby\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ScalyrAPIKey\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ScalyrImage\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ScalyrServerURL\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ScalyrCPURequest\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ScalyrMemoryRequest\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ScalyrCPULimit\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ScalyrMemoryLimit\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupSchedule\": \"30 00 * * *\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupDockerImage\": \"registry.opensource.zalan.do/acid/logical-backup:v1.7.0\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupProvider\": \"s3\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupS3Bucket\": \"my-bucket-url\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupS3Region\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupS3Endpoint\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupS3AccessKeyID\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupS3SecretAccessKey\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupS3SSE\": \"AES256\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupGoogleApplicationCredentials\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogicalBackupJobPrefix\": \"logical-backup-\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"NumberOfInstances\": 2," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"Schema\": \"pooler\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"User\": \"pooler\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"Image\": \"registry.opensource.zalan.do/acid/pgbouncer:master-18\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"Mode\": \"transaction\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MaxDBConnections\": 60," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ConnectionPoolerDefaultCPURequest\": \"500m\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ConnectionPoolerDefaultMemoryRequest\": \"100Mi\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ConnectionPoolerDefaultCPULimit\": \"1\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ConnectionPoolerDefaultMemoryLimit\": \"100Mi\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"WatchedNamespace\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"KubernetesUseConfigMaps\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EtcdHost\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DockerImage\": \"registry.opensource.zalan.do/acid/spilo-13:2.1-p1\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SidecarImages\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SidecarContainers\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodServiceAccountName\": \"postgres-operator-pod\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodServiceAccountDefinition\": \"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"ServiceAccount\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-pod\\\"}}\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodServiceAccountRoleBindingDefinition\": \"{\\\"apiVersion\\\":\\\"rbac.authorization.k8s.io/v1\\\",\\\"kind\\\":\\\"RoleBinding\\\",\\\"metadata\\\":{\\\"name\\\":\\\"postgres-operator-pod\\\"},\\\"roleRef\\\":{\\\"apiGroup\\\":\\\"rbac.authorization.k8s.io\\\",\\\"kind\\\":\\\"ClusterRole\\\",\\\"name\\\":\\\"postgres-operator-pod\\\"},\\\"subjects\\\":[{\\\"kind\\\":\\\"ServiceAccount\\\",\\\"name\\\":\\\"postgres-operator-pod\\\"}]}\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MasterPodMoveTimeout\": 1200000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DbHostedZone\": \"db.example.com\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"AWSRegion\": \"eu-central-1\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"WALES3Bucket\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"LogS3Bucket\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"KubeIAMRole\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"WALGSBucket\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"GCPCredentials\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"WALAZStorageAccount\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"AdditionalSecretMount\": \"\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"AdditionalSecretMountPath\": \"/meta/credentials\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableEBSGp3Migration\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableEBSGp3MigrationMaxSize\": 1000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"DebugLogging\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableDBAccess\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableTeamsAPI\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableTeamSuperuser\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"TeamAdminRole\": \"admin\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"RoleDeletionSuffix\": \"_deleted\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableTeamMemberDeprecation\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableAdminRoleForUsers\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnablePostgresTeamCRD\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnablePostgresTeamCRDSuperusers\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableMasterLoadBalancer\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableReplicaLoadBalancer\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"CustomServiceAnnotations\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"CustomPodAnnotations\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnablePodAntiAffinity\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodAntiAffinityTopologyKey\": \"kubernetes.io/hostname\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"StorageResizeMode\": \"pvc\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableLoadBalancer\": null," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ExternalTrafficPolicy\": \"Cluster\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MasterDNSNameFormat\": \"{cluster}.{team}.{hostedzone}\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ReplicaDNSNameFormat\": \"{cluster}-repl.{team}.{hostedzone}\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PDBNameFormat\": \"postgres-{cluster}-pdb\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnablePodDisruptionBudget\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableInitContainers\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableSidecars\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"Workers\": 8," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"APIPort\": 8080," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"RingLogLines\": 100," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ClusterHistoryEntries\": 1000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"TeamAPIRoleConfiguration\": {" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="      \"log_statement\": \"all\"" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   }," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodTerminateGracePeriod\": 300000000000," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PodManagementPolicy\": \"ordered_ready\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"ProtectedRoles\": [" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="      \"admin\"" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   ]," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"PostgresSuperuserTeams\": [" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="      \"postgres_superusers\"" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   ]," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"SetMemoryRequestToLimit\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableLazySpiloUpgrade\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableCrossNamespaceSecret\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnablePgVersionEnvVar\": true," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"EnableSpiloWalPathCompat\": false," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MajorVersionUpgradeMode\": \"off\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"MinimalMajorVersion\": \"9.5\"," pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="   \"TargetMajorVersion\": \"13\"" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="}" pkg=controller
time="2021-09-03T11:41:42Z" level=debug msg="acquiring initial list of clusters" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="no clusters running" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="started working in background" pkg=controller
time="2021-09-03T11:41:42Z" level=info msg="listening on :8080" pkg=apiserver
time="2021-09-03T11:41:42Z" level=debug msg="new node has been added: /avalon ()" pkg=controller
time="2021-09-03T11:41:42Z" level=debug msg="new node has been added: /ida ()" pkg=controller
time="2021-09-03T11:41:42Z" level=debug msg="new node has been added: /othala ()" pkg=controller
time="2021-09-03T11:42:05Z" level=info msg="ADD event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-09-03T11:42:05Z" level=info msg="creating a new Postgres cluster" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
time="2021-09-03T11:42:05Z" level=warning msg="master is not running, generated master endpoint does not contain any addresses" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=info msg="endpoint \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=info msg="master service \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=debug msg="No load balancer created for the replica service" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=info msg="replica service \"default/acid-minimal-cluster-repl\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=debug msg="team API is disabled" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=info msg="users have been initialized" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=info msg="syncing secrets" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=debug msg="created new secret default/zalando.acid-minimal-cluster.credentials.postgresql.acid.zalan.do, namespace: default, uid: d0f9fb76-6e40-4768-93a0-a6319907ff4d" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=debug msg="created new secret default/standby.acid-minimal-cluster.credentials.postgresql.acid.zalan.do, namespace: default, uid: 54d600ad-554e-419b-a785-bfb8385fa725" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:05Z" level=debug msg="created new secret default/postgres.acid-minimal-cluster.credentials.postgresql.acid.zalan.do, namespace: default, uid: 573c5160-60fd-491f-952a-326d665ff875" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:06Z" level=debug msg="created new secret default/foo-user.acid-minimal-cluster.credentials.postgresql.acid.zalan.do, namespace: default, uid: 7df319f2-2df3-4c52-81fb-eaa077d9a519" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:06Z" level=info msg="secrets have been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
W0903 11:42:06.248418       1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
time="2021-09-03T11:42:06Z" level=info msg="pod disruption budget \"default/postgres-acid-minimal-cluster-pdb\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:06Z" level=debug msg="created new statefulset \"default/acid-minimal-cluster\", uid: \"37badba5-2604-412c-b1f5-ea340986d775\"" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:06Z" level=info msg="statefulset \"default/acid-minimal-cluster\" has been successfully created" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:06Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:42:39Z" level=debug msg="Waiting for 2 pods to become ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:52:36Z" level=error msg="failed to create cluster: pod labels error: still failing after 200 retries" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2021-09-03T11:52:36Z" level=error msg="could not create cluster: pod labels error: still failing after 200 retries" cluster-name=default/acid-minimal-cluster pkg=controller worker=0

The error about the OperatorConfiguration CRD at the start can be ignored as far as I can tell. The Helm chart properly deployed the CRDs and the set configuration values are picked up properly so as far as I can tell that's a red herring.

Database Pod log
The pod log (for both instances of the minimal manifest) look like the following:

2021-09-03 11:42:36,710 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2021-09-03 11:42:38,722 - bootstrapping - INFO - Could not connect to 169.254.169.254, assuming local Docker setup
2021-09-03 11:42:38,726 - bootstrapping - INFO - No meta-data available for this provider
2021-09-03 11:42:38,727 - bootstrapping - INFO - Looks like your running local
2021-09-03 11:42:38,889 - bootstrapping - INFO - Configuring standby-cluster
2021-09-03 11:42:38,890 - bootstrapping - INFO - Configuring pam-oauth2
2021-09-03 11:42:38,890 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2021-09-03 11:42:38,890 - bootstrapping - INFO - Configuring patroni
2021-09-03 11:42:38,912 - bootstrapping - INFO - Writing to file /run/postgres.yml
2021-09-03 11:42:38,913 - bootstrapping - INFO - Configuring certificate
2021-09-03 11:42:38,913 - bootstrapping - INFO - Generating ssl certificate
2021-09-03 11:42:39,067 - bootstrapping - INFO - Configuring bootstrap
2021-09-03 11:42:39,067 - bootstrapping - INFO - Configuring pgbouncer
2021-09-03 11:42:39,067 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2021-09-03 11:42:39,067 - bootstrapping - INFO - Configuring crontab
2021-09-03 11:42:39,068 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2021-09-03 11:42:39,068 - bootstrapping - INFO - Configuring log
2021-09-03 11:42:39,068 - bootstrapping - INFO - Configuring wal-e
2021-09-03 11:42:39,068 - bootstrapping - INFO - Configuring pgqd
2021-09-03 11:42:40,382 INFO: Selected new K8s API server endpoint https://10.100.1.1:6443
2021-09-03 11:42:40,413 INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-09-03 11:42:40,415 INFO: Lock owner: None; I am acid-minimal-cluster-0
2021-09-03 11:42:40,459 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /home/postgres/pgdata/pgroot/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /usr/lib/postgresql/13/bin/pg_ctl -D /home/postgres/pgdata/pgroot/data -l logfile start

2021-09-03 11:42:41,699 INFO: postmaster pid=80
2021-09-03 11:42:41 UTC [80]: [1-1] 61320a31.50 0     LOG:  Auto detecting pg_stat_kcache.linux_hz parameter...
/var/run/postgresql:5432 - no response
2021-09-03 11:42:41 UTC [80]: [2-1] 61320a31.50 0     LOG:  pg_stat_kcache.linux_hz is set to 1000000
2021-09-03 11:42:42,713 ERROR: postmaster is not running
2021-09-03 11:42:42,722 INFO: removing initialize key after failed attempt to bootstrap the cluster
2021-09-03 11:42:42,734 INFO: renaming data directory to /home/postgres/pgdata/pgroot/data_2021-09-03-11-42-42
Traceback (most recent call last):
  File "/usr/local/bin/patroni", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 171, in main
    return patroni_main()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 139, in patroni_main
    abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 100, in abstract_main
    controller.run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 109, in run
    super(Patroni, self).run()
  File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 59, in run
    self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/__init__.py", line 112, in _run_cycle
    logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1471, in run_cycle
    info = self._run_cycle()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1345, in _run_cycle
    return self.post_bootstrap()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1238, in post_bootstrap
    self.cancel_initialization()
  File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1231, in cancel_initialization
    raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
/run/service/patroni: finished with code=1 signal=0
/run/service/patroni: sleeping 30 seconds

And then it repeats the last part over and over starting from ... INFO: Selected new K8s API server endpoint https://10.100.1.1:6443 to /run/service/patroni: sleeping ... seconds.

Postgres Logs
None? All files in ${PGLOG} (/home/postgres/pgdata/pgroot/pg_log) are empty and the path ${LOG_ENV_DIR} (/run/etc/log.d/env) does not exist.

Database Pod postgres.yml
The postgres.yml (and .config/patroni/patronictl.yaml which is identical) inside the pod (0) looks like:

bootstrap:
  dcs:
    loop_wait: 10
    maximum_lag_on_failover: 33554432
    postgresql:
      parameters:
        archive_mode: 'on'
        archive_timeout: 1800s
        autovacuum_analyze_scale_factor: 0.02
        autovacuum_max_workers: 5
        autovacuum_vacuum_scale_factor: 0.05
        checkpoint_completion_target: 0.9
        hot_standby: 'on'
        log_autovacuum_min_duration: 0
        log_checkpoints: 'on'
        log_connections: 'on'
        log_disconnections: 'on'
        log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
        log_lock_waits: 'on'
        log_min_duration_statement: 500
        log_statement: ddl
        log_temp_files: 0
        max_connections: 1000
        max_replication_slots: 10
        max_wal_senders: 10
        tcp_keepalives_idle: 900
        tcp_keepalives_interval: 100
        track_functions: all
        wal_level: hot_standby
        wal_log_hints: 'on'
      use_pg_rewind: true
      use_slots: true
    retry_timeout: 10
    ttl: 30
  initdb:
  - auth-host: md5
  - auth-local: trust
  post_init: /scripts/post_init.sh "zalandos"
  users:
    zalandos:
      options:
      - CREATEDB
      - NOLOGIN
      password: ''
kubernetes:
  bypass_api_service: true
  labels:
    application: spilo
  pod_ip: 10.50.10.38
  ports:
  - name: postgresql
    port: 5432
  role_label: spilo-role
  scope_label: cluster-name
  use_endpoints: true
postgresql:
  authentication:
    replication:
      password: FbvLsqKd6ERBQ4McuNfOu69cCYuiGfEo8m5q86o563K0GdviGdtqACCqxKT7F8NJ
      username: standby
    superuser:
      password: BTUPr6Gfw5SAQOuQL7BIhvVpTmX7edKcr48qMy6uau41eBbfg1ATAOwAqNFU6lU5
      username: postgres
  basebackup_fast_xlog:
    command: /scripts/basebackup.sh
    retries: 2
  bin_dir: /usr/lib/postgresql/13/bin
  callbacks:
    on_role_change: /scripts/on_role_change.sh zalandos true
  connect_address: 10.50.10.38:5432
  create_replica_method:
  - basebackup_fast_xlog
  data_dir: /home/postgres/pgdata/pgroot/data
  listen: '*:5432'
  name: acid-minimal-cluster-0
  parameters:
    archive_command: /bin/true
    bg_mon.history_buckets: 120
    bg_mon.listen_address: '::'
    extwlist.custom_path: /scripts
    extwlist.extensions: btree_gin,btree_gist,citext,hstore,intarray,ltree,pgcrypto,pgq,pg_trgm,postgres_fdw,tablefunc,uuid-ossp,hypopg,timescaledb,pg_partman
    log_destination: csvlog
    log_directory: ../pg_log
    log_file_mode: '0644'
    log_filename: postgresql-%u.log
    log_rotation_age: 1d
    log_truncate_on_rotation: 'on'
    logging_collector: 'on'
    pg_stat_statements.track_utility: 'off'
    shared_buffers: 51564MB
    shared_preload_libraries: bg_mon,pg_stat_statements,pgextwlist,pg_auth_mon,set_user,timescaledb,pg_cron,pg_stat_kcache
    ssl: 'on'
    ssl_cert_file: /run/certs/server.crt
    ssl_key_file: /run/certs/server.key
  pg_hba:
  - local   all             all                                   trust
  - hostssl all             +zalandos    127.0.0.1/32       pam
  - host    all             all                127.0.0.1/32       md5
  - hostssl all             +zalandos    ::1/128            pam
  - host    all             all                ::1/128            md5
  - local   replication     standby                    trust
  - hostssl replication     standby all                md5
  - hostnossl all           all                all                reject
  - hostssl all             +zalandos    all                pam
  - hostssl all             all                all                md5
  pgpass: /run/postgresql/pgpass
  use_unix_socket: true
  use_unix_socket_repl: true
restapi:
  connect_address: 10.50.10.38:8008
  listen: :8008
scope: acid-minimal-cluster

What I noticed here is that the bootstrap.users section does not seem to match up with the minimal manifest example spec.users. Tracing this back to the Pod deployment, I see a variable SPILO_CONFIGURATION set with that same mismatch compared to the minimal manifest definition but I don't know if that is related or not.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the issue with the User Guide minimal manifest and the shown Helm values, then inspect the created cluster’s Kubernetes status and events alongside the operator logs. Pay particular attention to the CRD schema error and service-account warning; done means the minimal PostgreSQL cluster reaches an online state.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.