st2.conf settings for RabbitMQ connection retrites/limits don't work

Due to absence of real health check endpoints in st2, for K8s deployment we don't want our app to re-connect on error, but instead exit service if connection to DB or MQ was broken and let service re-start and re-schedule via K8s algorithms.

There are some settings present in st2.conf around MQ connection retries which don't work:

[messaging]
# How many times should we retry connection before failing.
connection_retries = 10
# How long should we wait between connection retries.
connection_retry_wait = 10000

However even with connection_retries = 0 services are re-connecting on RabbitMQ connection failures.

2019-08-21 12:25:22,915 WARNING [-] Connection to broker lost, trying to re-establish connection...
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/mixins.py", line 175, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/mixins.py", line 197, in consume
    conn.drain_events(timeout=safety_interval)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 315, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 500, in drain_events
    while not self.blocking_read(timeout):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 506, in blocking_read
    return self.on_inbound_frame(frame)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 510, in on_inbound_method
    method_sig, payload, content,
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 126, in dispatch_method
    listener(*args)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 637, in _on_close
    self._x_close_ok()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 652, in _x_close_ok
2019-08-21 12:25:22,919 WARNING [-] Connection to broker lost, trying to re-establish connection...
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/mixins.py", line 175, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/mixins.py", line 197, in consume
    conn.drain_events(timeout=safety_interval)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 315, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 103, in drain_events
    return connection.drain_events(**kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 500, in drain_events
    while not self.blocking_read(timeout):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 506, in blocking_read
    return self.on_inbound_frame(frame)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 510, in on_inbound_method
    method_sig, payload, content,
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 126, in dispatch_method
    listener(*args)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 637, in _on_close
    self._x_close_ok()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 652, in _x_close_ok
    self.send_method(spec.Connection.CloseOk, callback=self._on_close_ok)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 51, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/method_framing.py", line 172, in write_frame
    write(view[:offset])
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/transport.py", line 288, in write
    self._write(s)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 401, in sendall
    tail = self.send(data, flags)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 395, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 382, in _send_loop
    self.send_method(spec.Connection.CloseOk, callback=self._on_close_ok)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 51, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/method_framing.py", line 172, in write_frame
    write(view[:offset])
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/transport.py", line 288, in write
    self._write(s)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 401, in sendall
    tail = self.send(data, flags)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 395, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 382, in _send_loop
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer
    return send_method(data, *args)
error: [Errno 104] Connection reset by peer

and

2019-08-21 12:25:23,052 WARNING [-] Broker connection error, trying again in 0 seconds: error(u'failed to resolve broker hostname',).
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/utils/functional.py", line 341, in retry_over_time
    return fun(*args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 275, in connect
    return self.connection
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 823, in connection
    self._connection = self._establish_connection()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 778, in _establish_connection
    conn = self.transport.establish_connection()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 307, in connect
    self.transport.connect()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/transport.py", line 81, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/transport.py", line 152, in _connect
    "failed to resolve broker hostname"))
error: failed to resolve broker hostname
2019-08-21 12:35:53,544 WARNING [-] Broker connection error, trying again in 4.0 seconds: error(111, 'ECONNREFUSED').
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/utils/functional.py", line 341, in retry_over_time
    return fun(*args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 275, in connect
    return self.connection
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 823, in connection
    self._connection = self._establish_connection()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 778, in _establish_connection
    conn = self.transport.establish_connection()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 307, in connect
    self.transport.connect()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/transport.py", line 81, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/transport.py", line 152, in _connect
    "failed to resolve broker hostname"))
error: [Errno 111] ECONNREFUSED

We'll need a way to configure real MQ retries and limits.