st2 services ignore MongoDB failures

Problem

When StackStorm services are already running and MongoDB backend suddenly goes down, no services usually detect connection error, report that in logs nor try to re-connect in a pro-active way. They keep running and "alive" as nothing happened.

Reproducing

1. Start StackStorm, follow the logs
2. Stop MongoDB
3. Notice that StackStorm services DGAF about any MongoDB connection issues

Bonus points go to st2api which even normally responds with empty results on HTTP requests.

It turns out that services start to report connection errors only when they're processing something and expecting a response from DB request. This can be tens of minutes, depending on st2 cluster workload.

This lazy behavior leads to a situation when we think service is working OK, while in fact it's just pretending and loosing incoming requests with no DB connection. https://github.com/StackStorm/st2/issues/4777 and #4020 is somewhat related.

Expected behavior

Good behavior if mongo client would verify connection in background loop and report back ASAP if there is an error in logs.

Ideally if it can support heartbeat setting in st2.conf and check pro-actively for DB connection aliveness:

• https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#heartbeatfrequencyms
• https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#heartbeatfrequencyms-defaults-to-10-seconds-or-60-seconds
• https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#what-s-the-point-of-periodic-monitoring

Note also https://api.mongodb.com/python/current/api/pymongo/monitoring.html
HeartbeatLogger and ConnectionPoolLogger

Start by reproducing the failure with StackStorm services running, then stop MongoDB and follow the service logs, especially st2api responses. Read the MongoDB heartbeat and monitoring references linked in the issue and inspect the st2.conf connection settings. Done means services detect and report the outage promptly and define the expected reconnection behavior.