Slow (30sec+) transitions between tasks when using loops (`with`, `items`, `concurrency`)

SUMMARY

It seems that when we using the with syntax:

  scan_servers:
    with:
      items: server in <% ctx(servers_to_check) %>
      concurrency: 1
    action: core.local
    input:
      cmd: "echo 'Hello <% item(server).id %>'"

transitions between two tasks are slow (30sec+). For comparison, transitions between two tasks when not using with is 0-2 seconds.

This is a problem for us because with a large number of tasks (e.g., if we want to check 500 servers), StackStorm will spend 500 * 30 = 15 000 seconds over 4 hours just waiting and doing nothing.

Questions:

• Is this expected/a known issue?
• Is there any setting or fix we can apply on our side to speed up this transition time?

STACKSTORM VERSION

st2 3.0.1, on Python 2.7.12

OS, environment, install method

Ubuntu 16.04, apt-get install stackstorm (added APT repository https://packagecloud.io/StackStorm/stable/ubuntu)

Steps to reproduce the problem

Here is a workflow we use to reproduce the problem. I replaced the real payload of the servers with fake JSON data from https://jsonplaceholder.typicode.com/.

version: 1.0

tasks:
  retrieve_server_list:
    action: core.local
    input:
      cmd: 'curl https://jsonplaceholder.typicode.com/comments'
    next:
      - when: <% succeeded() %>
        publish:
          - servers_to_check: <% result().stdout %>
        do:
          - scan_servers

  scan_servers:
    with:
      items: server in <% ctx(servers_to_check) %>
      concurrency: 1
    action: core.local
    input:
      cmd: "echo 'Hello <% item(server).id %>'"

Here we can see the (~ 30 seconds) in the logs as well:

2020-03-10 15:19:35,950 140241378099760 INFO workflows [-] [5e67af6b2c5a45070737a8dc] Identifying next set (0) of tasks after completion of task “scan_servers”, route “0".
2020-03-10 15:20:00,839 140241378099760 INFO workflows [-] [5e67af6b2c5a45070737a8dc] Identified the following set of tasks to execute next: scan_servers (route 0)

Expected Results

The transitions between tasks should take the same time as when not using with (0-2 seconds).

Actual Results

The transitions between tasks take 30sec+.

No source file or test is named. Start by running the supplied StackStorm workflow on the stated version and environment, compare loop transitions with non-with transitions, and use the provided workflow logs to locate where the 30-second delay occurs. Done means identifying the cause and confirming transitions approach the expected 0–2 seconds.