zalando / zalando/postgres-operator
Bug: operator fails to scaledown when -0 instance is failing
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
Please, answer some short questions which should help us to understand your problem / question better?
- image: registry.opensource.zalan.do/acid/postgres-operator:v1.8.2
- Bare metal k8s
- Production: Yes
- Bug
We currently have a postgresql resource that was set to 2 instances, and -1 is running and active. -0 won't start - fails with these errors:
2023-01-26 08:04:28,301 INFO: master_timeline=2
2023-01-26 08:04:28,304 INFO: master: history=1 5/ED0000A0 no recovery target specified
2023-01-26 08:04:28,304 INFO: Lock owner: mattermost-pgsql-1; I am mattermost-pgsql-0
2023-01-26 08:04:28,304 INFO: starting as a secondary
2023-01-26 08:04:28,469 INFO: postmaster pid=6388
/var/run/postgresql:5432 - no response
[CUT - pg starting up fine]
2023-01-26 08:04:28 UTC [6388]: [8-1] 63d2340c.18f4 0 HINT: Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - rejecting connections
/var/run/postgresql:5432 - no response
2023-01-26 08:04:38,224 INFO: Lock owner: mattermost-pgsql-1; I am mattermost-pgsql-0
2023-01-26 08:04:38,225 INFO: failed to start postgres
2023-01-26 08:04:48,227 WARNING: Postgresql is not running.
``
and I then figured, lets scale it down to just 1 instance - edited the postgresql object to 1, and operator then says:
-```
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"+ replicas: 1,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- port: 8008,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- scheme: HTTP","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"+ port: 8008","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- terminationMessagePath: /dev/termination-log,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- terminationMessagePolicy: File,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- restartPolicy: Always,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- dnsPolicy: ClusterFirst,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- serviceAccount: postgres-pod,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- },","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- schedulerName: default-scheduler","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"+ }","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- kind: PersistentVolumeClaim,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- apiVersion: v1,","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- status: {","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- phase: Pending","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- }","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"+ status: {}","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- },","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"- revisionHistoryLimit: 10","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"+ }","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"info","msg":"reason: new statefulset's number of replicas does not match the current one","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"debug","msg":"making POST http request: http://172.20.30.21:8008/failover","pkg":"cluster","time":"2023-01-26T07:53:08Z","worker":3}
{"cluster-name":"mattermost/mattermost-pgsql","level":"warning","msg":"could not scale down: could not failover: patroni returned 'failover is not possible: no good candidates have been found'","pkg":"cluster","time":"2023-01-26T07
So it seems the operator really WANTS to shutdown the -1 instance.. but as -0 is broken (for a reason I have yet to figure) - it can't - and thus is in a deadlock :(
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the scale-down through the postgres operator with one instance unhealthy, then trace the operator's StatefulSet replica handling and the Patroni /failover request. Done means the cluster reaches the requested replica count without deadlocking when the target instance cannot start or no failover candidate is available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, postgresql
- Domain
- databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100