zalando / zalando/postgres-operator

Sidecar without image specification causes all PostgreSQL pods to shutdown

Open
#2,966 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
5.2k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
3

Description

  • Which image of the operator are you using? ghcr.io/zalando/postgres-operator:v1.14.0
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
  • Are you running Postgres Operator in production? yes
  • Type of issue? Bug report

It's possible to add a sidecar specification without providing any image (by mistake) for that additional container.
This leads to a cascading failure where all cluster instances are shut down sequentially during sync intervals (30 minutes) until all pods are terminated, resulting in complete PostgreSQL unavailability.

New pods cannot start due to the Kubernetes error: "create Pod ... in StatefulSet ... failed error: Pod ... is invalid: spec.containers[2].image: Required value"

The operator logs show this -

{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"performing rolling update","pkg":"cluster","time":"2025-10-20T09:10:42Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"there are 2 pods in the cluster to recreate","pkg":"cluster","time":"2025-10-20T09:10:42Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"subscribing to pod \"dev-mdb-mq-clusters/test-no-image-1\"","pkg":"cluster","time":"2025-10-20T09:10:42Z","worker":2}
...
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"unsubscribing from pod \"dev-mdb-mq-clusters/test-no-image-1\" events","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"error","msg":"could not sync statefulsets: could not recreate pods: could not recreate replica pod \"dev-mdb-mq-clusters/test-no-image-1\": pod label wait timeout","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"syncing roles","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"db ready status: true","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"closing database connection","pkg":"cluster","time":"2025-10-20T09:20:52Z","worker":2}

Next operator sync kills last available pod -

{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"found rolling update flag on pod \"test-no-image-0\"","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"1 / 1 pod(s) still need to be rotated","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"performing rolling update","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"there are 1 pods in the cluster to recreate","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"warning","msg":"cannot perform switch over before re-creating the pod: no replicas","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"info","msg":"recreating old master pod \"dev-mdb-mq-clusters/test-no-image-0\"","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"subscribing to pod \"dev-mdb-mq-clusters/test-no-image-0\"","pkg":"cluster","time":"2025-10-20T09:36:13Z","worker":2}
...
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"debug","msg":"unsubscribing from pod \"dev-mdb-mq-clusters/test-no-image-0\" events","pkg":"cluster","time":"2025-10-20T09:46:24Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"warning","msg":"error while syncing cluster state: could not sync statefulsets: could not recreate pods: could not recreate old master pod \"dev-mdb-mq-clusters/test-no-image-0\": pod label wait timeout","pkg":"cluster","time":"2025-10-20T09:46:24Z","worker":2}
{"cluster-name":"dev-mdb-mq-clusters/test-no-image","level":"error","msg":"could not sync cluster: could not sync statefulsets: could not recreate pods: could not recreate old master pod \"dev-mdb-mq-clusters/test-no-image-0\": pod label wait timeout","pkg":"controller","time":"2025-10-20T09:46:25Z","worker":2}

The operator should validate the StatefulSet configuration before triggering updates.
At minimum, the sidecar image field should be marked as required in the CRD validation schema.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the CRD validation schema and the StatefulSet configuration or update path mentioned in the report. Reproduce a sidecar with no image, verify that validation rejects it before a rolling update, and add or update coverage for the invalid configuration and the resulting safe behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, postgresql
Domain
databases, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.