st2 services should log connection to [coordination] backend

st2 services log error if [coordination] connection was unsuccessful, which is great:

2019-08-30 11:19:54,609 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2019-08-30 11:19:54,609 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2019-08-30 11:19:54,610 DEBUG [-] Using logging config: /etc/st2/logging.api.gunicorn.conf
2019-08-30 11:19:54,622 INFO [-] Connecting to database "st2" @ "ignorant-manta-mongodb-ha-0.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-1.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-2.ignorant-manta-mongodb-ha:27017 (replica set)" as user "admin".
2019-08-30 11:19:54,797 INFO [-] Successfully connected to database "st2" @ "ignorant-manta-mongodb-ha-0.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-1.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-2.ignorant-manta-mongodb-ha:27017 (replica set)" as user "admin".
2019-08-30 11:19:56,138 INFO [-] (PID=1) ST2 API is serving on http://0.0.0.0:9101.
2019-08-30 11:19:56,138 INFO [-] Creating st2api: StackStorm v3.2dev as OpenAPI app.
2019-08-30 11:19:57,063 INFO [-] Connected to amqp://admin:**@ignorant-manta-rabbitmq-ha-discovery:5672//
2019-08-30 11:19:57,111 ERROR [-] (PID=1) ST2 API quit due to exception.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/cmd/api.py", line 81, in main
    return _run_server()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/cmd/api.py", line 70, in _run_server
    wsgi.server(sock, app.setup_app(), custom_pool=worker_pool, log=LOG, log_output=False)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/app.py", line 79, in setup_app
    router.add_spec(spec, transforms=transforms)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/router.py", line 211, in add_spec
    __import__(module_name)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/controllers/v1/keyvalue.py", line 436, in <module>
    key_value_pair_controller = KeyValuePairController()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2api/controllers/v1/keyvalue.py", line 61, in __init__
    self._coordinator = coordination.get_coordinator()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/services/coordination.py", line 226, in get_coordinator
    COORDINATOR = coordinator_setup(start_heart=start_heart)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/services/coordination.py", line 199, in coordinator_setup
    coordinator.start(start_heart=start_heart)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/tooz/coordination.py", line 426, in start
    self._start()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/tooz/drivers/etcd.py", line 252, in _start
    encodeutils.exception_to_unicode(e))
ToozConnectionError: HTTPConnectionPool(host='etcd-cluster-client', port=2379): Max retries exceeded with url: /v2/stats/self (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fca9effa850>: Failed to establish a new connection: [Errno -2] No address found',))

However it doesn't log any message if connection was successful to [coordination] backend:

$ kubectl logs ignorant-manta-st2api-fd58c5b8f-4x8gn
2019-08-30 11:22:12,899 DEBUG [-] Using Python: 2.7.12 (/opt/stackstorm/st2/bin/python)
2019-08-30 11:22:12,900 DEBUG [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
2019-08-30 11:22:12,900 DEBUG [-] Using logging config: /etc/st2/logging.api.gunicorn.conf
2019-08-30 11:22:12,910 INFO [-] Connecting to database "st2" @ "ignorant-manta-mongodb-ha-0.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-1.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-2.ignorant-manta-mongodb-ha:27017 (replica set)" as user "admin".
2019-08-30 11:22:13,194 INFO [-] Successfully connected to database "st2" @ "ignorant-manta-mongodb-ha-0.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-1.ignorant-manta-mongodb-ha:27017,ignorant-manta-mongodb-ha-2.ignorant-manta-mongodb-ha:27017 (replica set)" as user "admin".
2019-08-30 11:22:14,768 INFO [-] (PID=1) ST2 API is serving on http://0.0.0.0:9101.
2019-08-30 11:22:14,768 INFO [-] Creating st2api: StackStorm v3.2dev as OpenAPI app.
2019-08-30 11:22:15,917 INFO [-] Connected to amqp://admin:**@ignorant-manta-rabbitmq-ha-discovery:5672//
2019-08-30 11:22:16,287 INFO [-] Connected to amqp://admin:**@ignorant-manta-rabbitmq-ha-discovery:5672//

It's preferred to log this for informational, consistency and user expectation reasons, as it happens for both MongoDB and RabbitMQ in logs. This will als make it clearer which services really rely on coordination.

Start in st2common/services/coordination.py, especially coordinator_setup and the coordinator start path shown in the traceback. Review how successful MongoDB and RabbitMQ connections are logged, then add an informational message for a successful [coordination] backend connection. Done means services log the successful coordination connection consistently with the other backends.