st2 ha: packs/register command doesn't execute sensors reload on every node

SUMMARY

I have HA installation of Stackstorm with three nodes and pipeline which deploys my packs over all the nodes. Same pipeline executes API call POST -> /api/v1/packs/register with all the packs and as far as I understand, this command causes sensors reloading. But this is not what happens all the time - sometimes sensor might be reloaded on some node and sometimes not. This leads to cases when:

• some node is running sensors with new code.
• some node is running sensors with old code, but actions with new code.

Such behavior leads to completely upredictable behavior of your workflow based on sensors and actions.

STACKSTORM VERSION

st2 3.3.0, on Python 3.6.9

OS, environment, install method

VM, Ubuntu 18.04, custom install.

Steps to reproduce the problem

• Run HA installation with 3 nodes
• Get some packs with sensors installed
• Register packs via API call POST -> /api/v1/packs/register
• Update sensors code
• Register packs via API call POST -> /api/v1/packs/register
• Check logs of st2sensorcontainer for word "reloaded".
  You may need to do that few times to catch the issue.

Expected Results

Single API call POST -> /api/v1/packs/register causes sensors reloading on every node in HA installation.

Actual Results

Sometimes one or two nodes continue running sensors (i can see it by updated log messages in sensor's code) with old code and logs of st2sensorcontainer doesn't contain anything about reloading

Start by tracing the POST /api/v1/packs/register path and how it triggers reloads in st2sensorcontainer across HA nodes. Use the three-node reproduction and sensor reload logs to identify why nodes are skipped; done means one registration reliably reloads sensors on every node and the behavior is covered by an appropriate test.