st2rulesengine can not connect rabbitmq if rabbitmq stoped, after a long time, rabbitmq started, but st2rulesengine will not connect to rabbitmq automatically

SUMMARY

st2rulesengine can not connect rabbitmq if rabbitmq stoped, after a long time(for example 1 minitutes), rabbitmq started,
but st2rulesengine will not connect to rabbitmq automatically and it wil generate
many error messages.

ISSUE TYPE

Pick one below and delete the rest:

• Feature Idea
  It needs to check the mechanism of retrying to connect to rabbitmq in st2.

STACKSTORM VERSION

Paste the output of st2 --version:
st2-2.10.1-1.x86_64

OS / ENVIRONMENT / INSTALL METHOD

[root@localhost services]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[root@localhost services]# rpm -qa|grep rabbitmq
rabbitmq-server-3.3.5-34.el7.noarch
[root@localhost site-packages]# rpm -qa|grep st2
clutter-gst2-2.0.18-1.el7.x86_64
st2mistral-2.10.1-1.x86_64
st2-2.10.1-1.x86_64
st2web-2.10.3-1.x86_64
st2chatops-2.10.1-1.x86_64

STEPS TO REPRODUCE

step 1: execute the command below:
systemctl stop rabbitmq-server

step 2:
wait for more than 1 minitue

step 3:
systemctl start rabbitmq-server

EXPECTED RESULTS

st2rulesengine can connect to rabbitmq

ACTUAL RESULTS

There are many error messages in /var/log/st2/st2rulesengine.log as below:

2019-06-26T01:55:34.323967586Z 2019-06-26 09:55:34,310 WARNING [-] Broker connection error, trying again in 32.0 seconds: error(104, 'ECONNRESET').
2019-06-26T01:55:34.324091907Z Traceback (most recent call last):
2019-06-26T01:55:34.324134529Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
2019-06-26T01:55:34.324146053Z return fun(*args, **kwargs)
2019-06-26T01:55:34.324149922Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/connection.py", line 261, in connect
2019-06-26T01:55:34.324154189Z return self.connection
2019-06-26T01:55:34.324157467Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
2019-06-26T01:55:34.324161032Z self._connection = self._establish_connection()
2019-06-26T01:55:34.324164311Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
2019-06-26T01:55:34.324168067Z conn = self.transport.establish_connection()
2019-06-26T01:55:34.324176706Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
2019-06-26T01:55:34.324185926Z conn.connect()
2019-06-26T01:55:34.324190109Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/amqp/connection.py", line 307, in connect
2019-06-26T01:55:34.324203446Z self.transport.connect()
2019-06-26T01:55:34.324211141Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/amqp/transport.py", line 81, in connect
2019-06-26T01:55:34.324215466Z self._connect(self.host, self.port, self.connect_timeout)
2019-06-26T01:55:34.324222524Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/amqp/transport.py", line 148, in _connect
2019-06-26T01:55:34.32423016Z "failed to resolve broker hostname"))
2019-06-26T01:55:34.324237945Z error: [Errno 104] ECONNRESET
2019-06-26T01:56:06.369686977Z 2019-06-26 09:56:06,368 WARNING [-] Broker connection error, trying again in 32.0 seconds: timeout('timed out',).
2019-06-26T01:56:06.369734352Z Traceback (most recent call last):
2019-06-26T01:56:06.369740725Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
2019-06-26T01:56:06.369745672Z return fun(*args, **kwargs)
2019-06-26T01:56:06.36974939Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/connection.py", line 261, in connect
2019-06-26T01:56:06.369753656Z return self.connection
2019-06-26T01:56:06.369757076Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
2019-06-26T01:56:06.369761378Z self._connection = self._establish_connection()
2019-06-26T01:56:06.369764678Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
2019-06-26T01:56:06.369768525Z conn = self.transport.establish_connection()
2019-06-26T01:56:06.369771731Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
2019-06-26T01:56:06.369797909Z conn.connect()
2019-06-26T01:56:06.369815888Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/amqp/connection.py", line 313, in connect
2019-06-26T01:56:06.369820711Z self.drain_events(timeout=self.connect_timeout)
2019-06-26T01:56:06.36982395Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/amqp/connection.py", line 500, in drain_events
2019-06-26T01:56:06.369827713Z while not self.blocking_read(timeout):
2019-06-26T01:56:06.369830971Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/amqp/connection.py", line 505, in blocking_read
2019-06-26T01:56:06.369835223Z frame = self.transport.read_frame()
2019-06-26T01:56:06.369838544Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/amqp/transport.py", line 250, in read_frame
2019-06-26T01:56:06.369842295Z frame_header = read(7, True)
2019-06-26T01:56:06.369845611Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/amqp/transport.py", line 430, in _read
2019-06-26T01:56:06.36984923Z s = recv(n - len(rbuf))
2019-06-26T01:56:06.369852523Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/greenio/base.py", line 349, in recv
2019-06-26T01:56:06.369856165Z return self._recv_loop(self.fd.recv, bufsize, flags)
2019-06-26T01:56:06.369859873Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/greenio/base.py", line 343, in _recv_loop
2019-06-26T01:56:06.369863549Z self._read_trampoline()
2019-06-26T01:56:06.36986681Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/greenio/base.py", line 314, in _read_trampoline
2019-06-26T01:56:06.369871031Z timeout_exc=socket.timeout("timed out"))
2019-06-26T01:56:06.369876484Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/greenio/base.py", line 203, in _trampoline
2019-06-26T01:56:06.369880204Z mark_as_closed=self._mark_as_closed)
2019-06-26T01:56:06.369883806Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/hubs/init.py", line 162, in trampoline
2019-06-26T01:56:06.369887472Z return hub.switch()
2019-06-26T01:56:06.369890556Z File "/opt/stackstorm/st2/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-06-26T01:56:06.369894354Z return self.greenlet.switch()
2019-06-26T01:56:06.369897684Z timeout: timed out

Reproduce the failure with the reported systemctl stop/start rabbitmq-server steps and inspect the st2rulesengine broker retry behavior alongside /var/log/st2/st2rulesengine.log. Trace the connection path shown in the Kombu, amqp, and eventlet stack traces; done means st2rulesengine reconnects automatically after RabbitMQ is restarted without repeated connection errors.