zalando / zalando/postgres-operator

changes in initContainer's command doesn't trigger statefulset sync

Open
#2,259 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

question wontfix
Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.8.1
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? aws k8s v1.23
  • Are you running Postgres Operator in production? yes
  • Type of issue? bug report

Hi,

We are using initContainer to tune some kernel settings for nodes running postgres. postgresql manifest looks like this:

apiVersion: acid.zalan.do/v1
kind: postgresql
...
spec:
  ...
  additionalVolumes:
  - mountPath: /host-sys
    name: host-sys
    volumeSource:
      hostPath:
        path: /sys
  initContainers:
  - command:
    - /bin/sh
    - -c
    - |-
      sysctl -w vm.dirty_background_bytes=1600000
      sysctl -w vm.dirty_bytes=3200000
      sysctl -w vm.swappiness=1
      sysctl -w kernel.sched_autogroup_enabled=0
      sysctl -w kernel.sched_migration_cost_ns=5000000
    image: busybox
    name: sysctl-container-revision-1
    securityContext:
      privileged: true
      runAsUser: 0
    volumeMounts:
    - mountPath: /host-sys
      name: host-sys

We noticed that changes in "command" section does not trigger update of statefulset. For example, change of first line to sysctl -w vm.dirty_background_bytes=1600001 produces following log in postgres-operator (and doesn't trigger "syncing statefulsets"):

time="2023-03-10T13:58:33Z" level=info msg="UPDATE event has been queued" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T13:58:33Z" level=info msg="update of the cluster started" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T13:58:33Z" level=debug msg="-    resourceVersion: 16999969," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="-    generation: 24," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+    resourceVersion: 17001340," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+    generation: 25," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="-          sysctl -w vm.dirty_background_bytes=1600000\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+          sysctl -w vm.dirty_background_bytes=1600001\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="postgresql major version unchanged or smaller, no changes needed" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="initialize users" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="volume claims do not require changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=debug msg="syncing statefulsets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=debug msg="making GET http request: http://10.104.14.184:8008/config" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="making GET http request: http://10.104.15.187:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="making GET http request: http://10.104.14.184:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="syncing roles" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="closing database connection" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="syncing connection pooler (master, replica) from (true, nil) to (true, nil)" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=info msg="cluster has been updated" cluster-name=island-published/island-postgres pkg=controller worker=0

To get your changes applied to sts you have to patch any other parameter of manifest, for example if I change initContainer's name to sysctl-container-revision-2 then i can see in operator's log that sts is being updated:

time="2023-03-10T14:01:27Z" level=info msg="UPDATE event has been queued" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T14:01:27Z" level=info msg="update of the cluster started" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T14:01:27Z" level=debug msg="-    resourceVersion: 17001458," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="-    generation: 25," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+    resourceVersion: 17002746," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+    generation: 26," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="-        name: sysctl-container-revision-1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+        name: sysctl-container-revision-2," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=info msg="postgresql major version unchanged or smaller, no changes needed" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="initialize users" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=info msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="volume claims do not require changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=debug msg="syncing statefulsets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="mark rolling update annotation for island-postgres-0: reason pod changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="mark rolling update annotation for island-postgres-1: reason pod changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="statefulset island-published/island-postgres is not in the desired state and needs to be updated" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            path: /sys," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            type: " cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+            path: /sys" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          name: sysctl-container-revision-1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+          name: sysctl-container-revision-2," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            sysctl -w vm.dirty_background_bytes=1600000\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+            sysctl -w vm.dirty_background_bytes=1600001\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          imagePullPolicy: IfNotPresent," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-              port: 8008," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-              scheme: HTTP" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+              port: 8008" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      restartPolicy: Always," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      serviceAccount: default," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      schedulerName: default-scheduler," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      apiVersion: v1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      status: {" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-        phase: Pending" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      }" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+      status: {}" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-  }," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+  }" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="reason: new statefulset initContainers's sysctl-container-revision-1 (index 0) name does not match the current one" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="updating statefulset" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="making GET http request: http://10.104.14.184:8008/config" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="making GET http request: http://10.104.15.187:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="making GET http request: http://10.104.14.184:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="performing rolling update" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=info msg="there are 2 pods in the cluster to recreate" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="subscribing to pod \"island-published/island-postgres-1\"" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=info msg="pod \"island-published/island-postgres-1\" has been recreated" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="unsubscribing from pod \"island-published/island-postgres-1\" events" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="making GET http request: http://10.104.14.184:8008/cluster" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=error msg="could not sync statefulsets: could not recreate pods: skipping switchover: no switchover candidate found" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="syncing roles" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="closing database connection" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="syncing connection pooler (master, replica) from (true, nil) to (true, nil)" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=info msg="cluster has been updated" cluster-name=island-published/island-postgres pkg=controller worker=0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the operator's logged "syncing statefulsets" path and comparing the behavior for initContainer command versus name changes. Add a regression test for command-only changes and verify that the StatefulSet is updated and pods receive the new command.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.