Intermittent psycopg2.errors.AdminShutdown errors in galaxy_ng dev env and pulp-all-in-one

Author: alikins (alikins)

Redmine Issue: 8782, https://pulp.plan.io/issues/8782

A few folks have ran into an issue where when starting up the services including galaxy_ng, that
all of the pulp services will start throwing errors like:

worker_1            | pulp [None]: rq.worker:ERROR: Worker rq:worker:1@440b6c94ceca: found an unhandled exception, quitting...
worker_1            | Traceback (most recent call last):
worker_1            |   File "/venv/lib64/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
worker_1            |     return self.cursor.execute(sql, params)
worker_1            | psycopg2.errors.AdminShutdown: terminating connection due to administrator command
worker_1            | server closed the connection unexpectedly
worker_1            | 	This probably means the server terminated abnormally
worker_1            | 	before or while processing the request.

I've seen it with ~master ish galaxy_ng with the galaxy_ng dev containers. In that scenario it seems to get triggered by starting up the ./compose up ui ui component.

bmbouter saw it at least once with the pulp-all-in-one container, but it is hard to reproduce.

But filing here for reference.

No source file or test is named. Start with the worker traceback and try reproducing the failure by running ./compose up ui in the galaxy_ng development containers, then compare with the pulp-all-in-one container. Done is not defined by the report; it would require identifying the cause and agreeing on a fix or documented resolution.