docker-library / docker-library/postgres
postgres in container with PID 1 aggregating orphaned processes leading to restarts & recovery
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 2.5k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
The postgres in docker hosted on K8S infrastructure with periodic health and rediness checks experiences regular restarts of the database.
Logs show 'postgres "server process <PID>exited with exit code 2" ' followed by restart and recovery of the database.
happening with a variety of stable postgres versions (12, 14, 16) on the same system
deployed using kubernetes, we monitor health with exec probe
livenessProbe:
exec:
command:
- pg_isready
- -U
- dmp-admin
- -d
- dmp-entity
timeoutSeconds: 1
as well as readiness probe
readinessProbe:
exec:
command:
- /bin/bash
- -c
- pg_isready -U dmp-admin -d dmp-entity && [ ! -f /var/lib/postgresql/backup/pgdump_backup.velero.sql ]
timeoutSeconds: 1
Investigation shows that the postgres db is fine - no exit 2 occurences.
Instead a regular health monitoring process that includes a /bin/bash -c pg_isready .. causes the problem.
Root cause is described here: https://www.cybertec-postgresql.com/en/docker-sudden-death-for-postgresql/ (thanks to laurentz albe)
The postgres in docker setup runs the default process (postgres) as root process of the container with PID 1.
The main postgres container is a health manager/monitor for all other spawned worker processes.
Sudden exits of worker processes lead to DB restart - remediating possible shared memory corruption.
However, orphaned other processes will get the root process as parent process (PID 1) being our postgres main entrypoint.
These processes get orphaned due to to timeout of the monitoring environment.
pg_isready is known to return exit code 2 when not able to connect.
Suggested remediation is in the referenced article: start the container using dum-init.
Example patch we deploy to remediate consists of installing the dumb-init package (using apt) and extending the entrypoint to use dumb-init as the main process (PID 1)
FROM postgres:12
RUN apt update && apt install -y dumb-init && apt clean
ENTRYPOINT ["/usr/bin/dumb-init", "docker-entrypoint.sh"]
CMD ["postgres"]
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported Dockerfile example, the docker-entrypoint.sh command, and the linked dumb-init article. The issue does not name a repository file or test, so first confirm whether the requested scope is an image entrypoint change; done should include a repository-supported remediation and validation that the probe-related restarts no longer occur.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes, postgres, shell
- Domain
- databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100