zalando / zalando/postgres-operator
Invalid data directory permissions after pod rescheduling
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.2k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 3
Description
Environment:
kubernetes 1.17.4
rook-ceph as persistent storage provider
postgres-operator 1.4.0 installed with helm chart
posgresql resource declared as follows:
kind: postgresql
metadata:
name: test-postgres
namespace: test
spec:
teamId: test
volume:
size: 50Gi
storageClass: rook-ceph-block
numberOfInstances: 2
users:
zalando:
- superuser
- createdb
service: []
databases:
db_1: service
db_2: service
postgresql:
version: "12"
parameters:
max_connections: "500"
Problem:
After pod with postgresql instance had been rescheduled to another node postgresql server doesn't start with following error:
2020-05-09 14:07:17 UTC [2507]: [2-1] 5eb6b915.9cb 0 DETAIL: Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
2020-05-09 14:07:17,085 INFO: postmaster pid=2507
/var/run/postgresql:5432 - no response
2020-05-09 14:07:17,116 WARNING: Postgresql is not running```
/home/postgres/pgdata/pgroot/data - have following permissions:
```# ls -alh pgdata/pgroot
total 16K
drwxrwsr-x 4 postgres postgres 4.0K Apr 29 11:34 .
drwxrwsrwx 4 root 1337 4.0K Apr 29 11:33 ..
drwxrws--- 19 postgres postgres 4.0K May 9 14:05 data
drwxrwsr-x 2 postgres postgres 4.0K May 5 20:49 pg_lo```
Workaround: change permissions to 750 by hands
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the pod rescheduling scenario with the listed Kubernetes, rook-ceph, postgres-operator, and PostgreSQL versions, then trace how the data directory permissions are set and preserved across nodes. Done means a rescheduled PostgreSQL pod starts without manually changing permissions and the relevant regression is covered by the project's available tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, helm, kubernetes, postgresql
- Domain
- databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100