Socket creation problem with Upgrading database (by switching to newer PostgreSQL image version)

Start with the upgrade README for the generated/9.6 image and inspect pg_upgrade_internal.log together with the referenced pg_upgrade_server.log. Reproduce the upgrade and trace the socket path, permissions, and process user shown in the failure. Done means the upgrade completes without the missing Unix-domain-socket error and the behavior is documented or covered by an appropriate project change.

Hello. This process "Upgrading database (by switching to newer PostgreSQL image version)" looks awesome and is a really cool idea. See this readme -> https://github.com/sclorg/postgresql-container/tree/generated/9.6

Unfortunately it does not work fully for me. It would appear there is a problem creating the socket.

The logs files from the upgrade are below.

sh-4.2$ more pg_upgrade_internal.log

---

pg_upgrade run on Mon Mar 2 15:19:46 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:20:58 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:22:17 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:24:13 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:28:06 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:29:18 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:30:56 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:32:49 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:34:46 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:36:35 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

---

pg_upgrade run on Mon Mar 2 15:39:46 2020

Performing Consistency Checks

Checking cluster versions ok

failure
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/lib/pgsql/data/.s.PGSQL.50432"?

could not connect to old postmaster started with the command:
"/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -b -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/pgsql/data'" st
art

I wonder is this a permissions issue or is postgresql already started. It is worth noting that normally the socket gets created in /tmp but during the upgrade process it attempts to create the socket in /var/lib/pgsql/data/.s.PGSQL.50432. I am not sure which user attempts to create the socket during the upgrade process.

Is it the uid below
sh-4.2$ id
uid=1001440000 gid=0(root) groups=0(root),2044,1001440000
sh-4.2$

Or the postgres user?

sh-4.2$ more /etc/passwd
root❌0:0:root:/root:/bin/bash
bin❌1:1:bin:/bin:/sbin/nologin
daemon❌2:2:daemon:/sbin:/sbin/nologin
adm❌3:4:adm:/var/adm:/sbin/nologin
lp❌4:7:lp:/var/spool/lpd:/sbin/nologin
sync❌5:0:sync:/sbin:/bin/sync
shutdown❌6:0:shutdown:/sbin:/sbin/shutdown
halt❌7:0:halt:/sbin:/sbin/halt
mail❌8:12:mail:/var/spool/mail:/sbin/nologin
operator❌11:0:operator:/root:/sbin/nologin
games❌12💯games:/usr/games:/sbin/nologin
ftp❌14:50:FTP User:/var/ftp:/sbin/nologin
nobody❌99:99:Nobody:/:/sbin/nologin
systemd-network❌192:192:systemd Network Management:/:/sbin/nologin
dbus❌81:81:System message bus:/:/sbin/nologin
default❌1001:0:Default Application User:/opt/app-root/src:/sbin/nologin
postgres❌26:26:PostgreSQL Server:/var/lib/pgsql:/bin/bash

sh-4.2$ ls -a /tmp
. .. .ICE-unix .Test-unix .X11-unix .XIM-unix .font-unix .s.PGSQL.5432 .s.PGSQL.5432.lock ks-script-OQQXsO yum.log
sh-4.2$

I have also raised a redhat support ticket for this issue (02599066). Any information/discussion much appreciated.