PgBouncer: cannot connect to server

Hi all,

Which image of the operator are you using? v1.7.0
Where do you run it - cloud or metal? Kubernetes or OpenShift? Scaleway Kapsule
Are you running Postgres Operator in production? Yes
Type of issue? pooler error

We have just migrated our new infrastructure into production, which is based on the operator Zalando. We have no worries about our test environment which is equivalent from a stack point of view, with only less resources at the hardware level.

But on the production environment we have sometime errors coming from PgBouncer. This crashes pods that maintain a database connection.

I asked Scaleway to verify that there was no problem on the network layer, but if someone here has already encountered this problem this could help us.

The connection pooler mode is "session".

Thank you and take care.

PgBouncer logs :

2022-07-06 01:00:05.562 UTC [1] WARNING S-0x562315c08280: db/db_writer_user@X.X.X.X:5432 got packet 'N' from server when not linked
2022-07-06 01:00:05.562 UTC [1] WARNING C-0x55b3534cd150: db/db_writer_user@X.X.X.X:46936 pooler error: server conn crashed?
2022-07-06 01:00:05.562 UTC [1] WARNING S-0x562315c07620: db/db_writer_user@X.X.X.X:5432 got packet 'N' from server when not linked
2022-07-06 01:00:05.563 UTC [1] WARNING C-0x562315ba34f0: db/db_reader_user@X.X.X.X:49014 pooler error: server conn crashed?
2022-07-06 01:00:05.564 UTC [1] WARNING S-0x55b3535316a0: db/db_writer_user@X.X.X.X:5432 got packet 'N' from server when not linked
2022-07-06 01:00:05.565 UTC [1] WARNING C-0x562315ba30d0: db/db_reader_user@X.X.X.X:42976 pooler error: server conn crashed?
2022-07-06 01:00:05.589 UTC [1] WARNING S-0x5625552e3200: db/pooler@X.X.X.X:5432 got packet 'N' from server when not linked
2022-07-06 01:00:05.590 UTC [1] WARNING tls_sbufio_recv: read failed: Connection reset by peer
2022-07-06 01:00:05.592 UTC [1] WARNING S-0x562315c07200: db/pooler@X.X.X.X:5432 got packet 'N' from server when not linked
2022-07-06 01:00:05.592 UTC [1] WARNING C-0x562315ba2470: db/db_writer_user@X.X.X.X:55426 pooler error: server conn crashed?
2022-07-06 01:00:05.594 UTC [1] WARNING tls_sbufio_recv: read failed: Connection reset by peer
2022-07-06 01:00:05.594 UTC [1] WARNING C-0x55b3534cc0d0: db/db_writer_user@X.X.X.X:46912 pooler error: server conn crashed?
2022-07-06 01:00:05.595 UTC [1] WARNING tls_sbufio_recv: read failed: Connection reset by peer
2022-07-06 01:00:05.595 UTC [1] WARNING S-0x55b35352f5a0: db/db_writer_user@X.X.X.X:5432 got packet 'N' from server when not linked
2022-07-06 01:00:05.595 UTC [1] WARNING tls_sbufio_recv: read failed: Connection reset by peer
2022-07-06 01:00:05.597 UTC [1] WARNING C-0x55b3534cdba0: db/db_writer_user@X.X.X.X:47812 pooler error: server conn crashed?
2022-07-06 01:00:05.689 UTC [1] WARNING tls_sbufio_recv: read failed: Connection reset by peer
2022-07-06 01:00:05.689 UTC [1] WARNING C-0x562315ba2ec0: db/db_writer_user@X.X.X.X:42014 pooler error: server conn crashed?
2022-07-06 01:00:05.893 UTC [1] ERROR S: login failed: FATAL: the database system is in recovery mode
2022-07-06 01:00:06.228 UTC [1] WARNING C-0x55b3534cbcb0: db/(nouser)@X.X.X.X:45886 pooler error: pgbouncer cannot connect to server
2022-07-06 01:00:06.242 UTC [1] ERROR S: login failed: FATAL: the database system is in recovery mode
2022-07-06 01:00:06.244 UTC [1] WARNING C-0x56255527ecb0: db/(nouser)@X.X.X.X:45880 pooler error: pgbouncer cannot connect to server
2022-07-06 01:00:06.859 UTC [1] WARNING TLS handshake error: handshake failed: Connection reset by peer
2022-07-06 01:00:27.102 UTC [1] WARNING C-0x55b3534cc910: db/db_reader_user@X.X.X.X:55040 taking connection from reserve_pool

Start with the PgBouncer logs in the issue and correlate the connection resets and “database system is in recovery mode” messages with the production PostgreSQL cluster on Scaleway Kapsule. Done means identifying a reproducible cause and documenting a verified remediation; the issue names no source files or tests.