Coordinator errors when installing packs and using Consul backend

SUMMARY

Using Consul as a coordinator backend, installing packs fails with errors.

STACKSTORM VERSION

# st2 --version
st2 3.5.0, on Python 3.6.8

OS, environment, install method

RockyLinux 8.4 installed per https://docs.stackstorm.com/install/rhel8.html
Consul v1.5.2 (I've also tried the latest 1.10.3 with the same errors).
python-consul2-0.1.5 (python-consul is no longer maintained).

Steps to reproduce the problem

• Start a Consul dev server

consul agent -dev

• Modify st2.conf to specify Consul as the coordinator

[coordination]
url = consul://127.0.0.1:8500

• Start ST2

st2ctl start

• Install a pack

st2 pack install st2

Expected Results

The pack should install

Actual Results

The install fails with errors:

# st2 pack install st2

For the "st2" pack, the following content will be registered:

actions   |  24
rules     |  0
sensors   |  0
aliases   |  12
triggers  |  0

Installation may take a while for packs with many items.

    [ succeeded ] init_task

id: 6165b63260a18b88e15ec783
action.ref: packs.install
parameters:
  packs:
  - st2
status: failed
start_timestamp: Tue, 12 Oct 2021 16:22:10 UTC
end_timestamp: Tue, 12 Oct 2021 16:22:11 UTC
log:
  - status: requested
    timestamp: '2021-10-12T16:22:10.930000Z'
  - status: scheduled
    timestamp: '2021-10-12T16:22:11.051000Z'
  - status: running
    timestamp: '2021-10-12T16:22:11.105000Z'
  - status: failed
    timestamp: '2021-10-12T16:22:11.905000Z'
result:
  errors:
  - message: Execution failed. See result for details.
    task_id: init_task
    type: error
  - message: 'TypeError: <lambda>() got an unexpected keyword argument ''retry_state'''
    route: 0
    task_id: init_task
    type: error
  - message: 'YaqlEvaluationException: Unable to resolve key ''packs_list'' in expression ''<% ctx().packs_list %>'' from context.'
    type: error
  - message: 'YaqlEvaluationException: Unable to resolve key ''message'' in expression ''<% ctx().message %>'' from context.'
    type: error
  - message: 'YaqlEvaluationException: Unable to resolve key ''conflict_list'' in expression ''<% ctx().conflict_list %>'' from context.'
    type: error
  - message: 'YaqlEvaluationException: Unable to resolve key ''warning_list'' in expression ''<% ctx().warning_list %>'' from context.'
    type: error
  output: null
+--------------------------+------------------------+-----------+-----------+-------------------------------+
| id                       | status                 | task      | action    | start_timestamp               |
+--------------------------+------------------------+-----------+-----------+-------------------------------+
| 6165b6339da5b6ea4fa6588a | succeeded (0s elapsed) | init_task | core.noop | Tue, 12 Oct 2021 16:22:11 UTC |
+--------------------------+------------------------+-----------+-----------+-------------------------------+

It looks like this is related to tenacity==7.0.0. If I downgrade it to tenacity==6.3.1 then I'm able to successfully install a pack:

# /opt/stackstorm/st2/bin/pip install --upgrade install tenacity==6.3.1
Collecting tenacity==6.3.1
  Downloading http://repository.overstock.com/repository/python-group/packages/tenacity/6.3.1/tenacity-6.3.1-py2.py3-none-any.whl (36 kB)
Requirement already satisfied: six>=1.9.0 in /opt/stackstorm/st2/lib/python3.6/site-packages (from tenacity==6.3.1) (1.13.0)
Collecting install
  Downloading http://repository.overstock.com/repository/python-group/packages/install/1.3.4/install-1.3.4-py3-none-any.whl (3.1 kB)
Installing collected packages: tenacity, install
  Attempting uninstall: tenacity
    Found existing installation: tenacity 7.0.0
    Uninstalling tenacity-7.0.0:
      Successfully uninstalled tenacity-7.0.0
Successfully installed install-1.3.4 tenacity-6.3.1
WARNING: You are using pip version 20.3.3; however, version 21.2.4 is available.
You should consider upgrading via the '/opt/stackstorm/st2/bin/python -m pip install --upgrade pip' command.

# st2ctl restart
Failed to stop st2chatops.service: Unit st2chatops.service not loaded.
^[[BFailed to start st2chatops.service: Unit st2chatops.service not found.
##### st2 components status #####
st2actionrunner PID: 83522
st2actionrunner PID: 83524
st2actionrunner PID: 83526
st2actionrunner PID: 83528
st2actionrunner PID: 83530
st2actionrunner PID: 83532
st2actionrunner PID: 83534
st2actionrunner PID: 83538
st2actionrunner PID: 83546
st2actionrunner PID: 83552
st2api PID: 83558
st2stream PID: 83564
st2stream PID: 83611
st2stream PID: 83613
st2stream PID: 83620
st2stream PID: 83621
st2auth PID: 83569
st2auth PID: 83658
st2garbagecollector PID: 83572
st2notifier PID: 83576
st2rulesengine PID: 83581
st2sensorcontainer PID: 83586
st2chatops is not running.
st2timersengine PID: 83595
st2workflowengine PID: 83599
st2scheduler PID: 83603

# st2 pack install st2

For the "st2" pack, the following content will be registered:

actions   |  24
rules     |  0
sensors   |  0
aliases   |  12
triggers  |  0

Installation may take a while for packs with many items.

    [ succeeded ] init_task
    [ succeeded ] download_pack
    [ succeeded ] make_a_prerun
    [ succeeded ] get_pack_dependencies
    [ succeeded ] check_dependency_and_conflict_list
    [ succeeded ] install_pack_requirements
    [ succeeded ] get_pack_warnings
    [ succeeded ] register_pack

+-------------+----------------------------------------+
| Property    | Value                                  |
+-------------+----------------------------------------+
| ref         | st2                                    |
| name        | st2                                    |
| description | StackStorm utility actions and aliases |
| version     | 2.0.1                                  |
| author      | StackStorm, Inc.                       |
+-------------+----------------------------------------+

Start by reproducing the Consul coordinator failure from st2.conf with tenacity 7.0.0, then compare it with tenacity 6.3.1 and trace the retry-related error during st2 pack install st2. Done means pack installation succeeds with the supported tenacity version and the regression is covered by a test.