Increase execution time for StackStorm workflows
Open
Nobody has claimed this yet.
performance
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 787
- PR merge metrics
- No merged PRs in 30d
Description
Issue Description
When I launch multiple workflows at the same time in StackStorm, I've noticed that the execution time of my workflows increases significantly.
STACKSTORM VERSION
st2 3.7.0, on Python 3.6.8
OS, environment, install method
Post what OS you are running this on, along with any other relevant information/
- e.g. one-line install, custom install, etc -->
- OS: Linux CentOS 3.10.0-1160.81.1.el7.x86_64
Steps to Reproduce
Launch multiple workflows at the same time in StackStorm
Observe the execution time of each workflow
Expected Behavior
The execution time of each workflow should remain constant, regardless of how many workflows are launched simultaneously.
Actual Behavior
The execution time of each workflow increases as more workflows are launched simultaneously.
Here the example of multiple launch of the same workflows, the exec time is almost 25 sec or 2 sec (due to a cancel check) for a normal behavior.
We can see, more I run workflow, more the exec time increase.
the issue seems to be that the workflow actions are getting stuck in a "pending" state, and we are not seeing any CPU overload in htop.
| succeeded (25s elapsed) | Fri, 17 Mar 2023 13:20:32 CET | Fri, 17 Mar 2023 13:20:57 CET |
| | | |
| succeeded (33s elapsed) | Fri, 17 Mar 2023 13:28:46 CET | Fri, 17 Mar 2023 13:29:19 CET |
| | | |
| succeeded (2s elapsed) | Fri, 17 Mar 2023 13:28:50 CET | Fri, 17 Mar 2023 13:28:52 CET |
| | | |
| succeeded (3s elapsed) | Fri, 17 Mar 2023 13:28:53 CET | Fri, 17 Mar 2023 13:28:56 CET |
| | | |
| succeeded (2s elapsed) | Fri, 17 Mar 2023 13:28:57 CET | Fri, 17 Mar 2023 13:28:59 CET |
| | | |
| succeeded (162s elapsed) | Fri, 17 Mar 2023 13:29:01 CET | Fri, 17 Mar 2023 13:31:43 CET |
| | | |
| succeeded (217s elapsed) | Fri, 17 Mar 2023 13:29:04 CET | Fri, 17 Mar 2023 13:32:41 CET |
| | | |
| succeeded (258s elapsed) | Fri, 17 Mar 2023 13:29:08 CET | Fri, 17 Mar 2023 13:33:26 CET |
| | | |
| succeeded (285s elapsed) | Fri, 17 Mar 2023 13:29:12 CET | Fri, 17 Mar 2023 13:33:57 CET |
| | | |
| succeeded (311s elapsed) | Fri, 17 Mar 2023 13:29:16 CET | Fri, 17 Mar 2023 13:34:27 CET |
| | | |
| succeeded (335s elapsed) | Fri, 17 Mar 2023 13:29:19 CET | Fri, 17 Mar 2023 13:34:54 CET |
| | | |
| succeeded (354s elapsed) | Fri, 17 Mar 2023 13:29:23 CET | Fri, 17 Mar 2023 13:35:17 CET |
| | | |
| succeeded (370s elapsed) | Fri, 17 Mar 2023 13:29:26 CET | Fri, 17 Mar 2023 13:35:36 CET |
| | | |
| succeeded (383s elapsed) | Fri, 17 Mar 2023 13:29:30 CET | Fri, 17 Mar 2023 13:35:53 CET |
| | | |
| succeeded (395s elapsed) | Fri, 17 Mar 2023 13:29:34 CET | Fri, 17 Mar 2023 13:36:09 CET |
| | | |
| succeeded (400s elapsed) | Fri, 17 Mar 2023 13:29:38 CET | Fri, 17 Mar 2023 13:36:18 CET |
| | | |
| succeeded (406s elapsed) | Fri, 17 Mar 2023 13:29:41 CET | Fri, 17 Mar 2023 13:36:27 CET |
| | | |
| succeeded (409s elapsed) | Fri, 17 Mar 2023 13:29:45 CET | Fri, 17 Mar 2023 13:36:34 CET |
| | | |
| succeeded (412s elapsed) | Fri, 17 Mar 2023 13:29:49 CET | Fri, 17 Mar 2023 13:36:41 CET |
| | | |
| succeeded (415s elapsed) | Fri, 17 Mar 2023 13:29:52 CET | Fri, 17 Mar 2023 13:36:47 CET |
| | | |
| succeeded (299s elapsed) | Fri, 17 Mar 2023 13:29:56 CET | Fri, 17 Mar 2023 13:34:55 CET |
| | | |
| succeeded (419s elapsed) | Fri, 17 Mar 2023 13:29:59 CET | Fri, 17 Mar 2023 13:36:58 CET |
| | | |
| succeeded (419s elapsed) | Fri, 17 Mar 2023 13:30:03 CET | Fri, 17 Mar 2023 13:37:02 CET |
| | | |
| succeeded (421s elapsed) | Fri, 17 Mar 2023 13:30:07 CET | Fri, 17 Mar 2023 13:37:08 CET |
| | | |
| succeeded (422s elapsed) | Fri, 17 Mar 2023 13:30:11 CET | Fri, 17 Mar 2023 13:37:13 CET |
| | | |
| succeeded (424s elapsed) | Fri, 17 Mar 2023 13:30:14 CET | Fri, 17 Mar 2023 13:37:18 CET |
| | | |
| succeeded (421s elapsed) | Fri, 17 Mar 2023 13:30:18 CET | Fri, 17 Mar 2023 13:37:19 CET |
| | | |
| succeeded (421s elapsed) | Fri, 17 Mar 2023 13:30:21 CET | Fri, 17 Mar 2023 13:37:22 CET |
| | | |
| succeeded (420s elapsed) | Fri, 17 Mar 2023 13:30:25 CET | Fri, 17 Mar 2023 13:37:25 CET |
| | | |
| succeeded (419s elapsed) | Fri, 17 Mar 2023 13:30:29 CET | Fri, 17 Mar 2023 13:37:28 CET |
| | | |
| succeeded (418s elapsed) | Fri, 17 Mar 2023 13:30:32 CET | Fri, 17 Mar 2023 13:37:30 CET |
| | | |
| succeeded (418s elapsed) | Fri, 17 Mar 2023 13:30:36 CET | Fri, 17 Mar 2023 13:37:34 CET |
| | | |
| succeeded (415s elapsed) | Fri, 17 Mar 2023 13:30:40 CET | Fri, 17 Mar 2023 13:37:35 CET |
| | | |
| succeeded (414s elapsed) | Fri, 17 Mar 2023 13:30:43 CET | Fri, 17 Mar 2023 13:37:37 CET |
| | | |
| succeeded (417s elapsed) | Fri, 17 Mar 2023 13:30:47 CET | Fri, 17 Mar 2023 13:37:44 CET |
| | | |
| succeeded (320s elapsed) | Fri, 17 Mar 2023 13:30:51 CET | Fri, 17 Mar 2023 13:36:11 CET |
| | | |
| succeeded (406s elapsed) | Fri, 17 Mar 2023 13:30:54 CET | Fri, 17 Mar 2023 13:37:40 CET |
| | | |
| succeeded (29s elapsed) | Fri, 17 Mar 2023 13:30:58 CET | Fri, 17 Mar 2023 13:31:27 CET |
| | | |
| succeeded (404s elapsed) | Fri, 17 Mar 2023 13:31:01 CET | Fri, 17 Mar 2023 13:37:45 CET |
| | | |
| succeeded (401s elapsed) | Fri, 17 Mar 2023 13:31:05 CET | Fri, 17 Mar 2023 13:37:46 CET |
| | | |
| succeeded (398s elapsed) | Fri, 17 Mar 2023 13:31:09 CET | Fri, 17 Mar 2023 13:37:47 CET |
| | | |
| succeeded (396s elapsed) | Fri, 17 Mar 2023 13:31:12 CET | Fri, 17 Mar 2023 13:37:48 CET |
| | | |
| succeeded (393s elapsed) | Fri, 17 Mar 2023 13:31:16 CET | Fri, 17 Mar 2023 13:37:49 CET |
| | | |
| succeeded (389s elapsed) | Fri, 17 Mar 2023 13:31:20 CET | Fri, 17 Mar 2023 13:37:49 CET |
| | | |
| succeeded (386s elapsed) | Fri, 17 Mar 2023 13:31:24 CET | Fri, 17 Mar 2023 13:37:50 CET |
| | | |
+--------------------------+-------------------------------+-------------------------------+
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported slowdown on StackStorm 3.7.0 with multiple simultaneous workflows and inspect why workflow actions remain pending. Compare execution times as concurrency increases and identify the relevant execution path before deciding on a fix. Done means concurrent workflows no longer experience the reported increasing delays.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100