st2rulesengine keep high CPU

SUMMARY

Provide a quick summary of your bug report.

STACKSTORM VERSION

Paste the output of st2 --version:

• st2 3.1.0, on Python 2.7.5

OS, environment, install method

Post what OS you are running this on, along with any other relevant information/

• e.g. Docker, Vagrant, Kubernetes, etc. Describe how you installed ST2
• e.g. one-line install, custom install, etc -->

• RHEL 7.6

• installed as instruction right here https://docs.stackstorm.com/install/rhel7.html

Steps to reproduce the problem

Show how to reproduce the problem, using a minimal test-case. Make sure to include any content
(pack content - workflows, actions, etc.) which are needed to reproduce the problem.

• define a rule for a webhook

---
    name: "sample_rule_with_webhook"
    pack: "mcm_chatops"
    description: "Sample rule dumping webhook payload to a file."
    enabled: true

    trigger:
        type: "core.st2.webhook"
        parameters:
            url: "cem-webhook"

    criteria:
        trigger.headers:
            type: "exists"
            pattern: "Content-Length"

    action:
        ref: "mcm_chatops.cem"
        parameters:
            headers: "{{trigger.headers}}"
            body:    "{{trigger.body}}"

• define an action

import re
import sys
import json
import requests
from datetime import datetime

from st2common.runners.base_action import Action

class CEMAction(Action):

    def run(self, headers, body):
        print('type(body)')
        print(type(body))
        print(body)

        print('type(headers)')
        print(type(headers))
        print(headers)

        if "eventsURL" in body:
            print(body["eventsURL"])

• and

---
    name: "cem"
    runner_type: "python-script"
    description: "query events from CEM"
    enabled: true
    entry_point: "cem.py"
    parameters:
        headers:
            type: "object"
            required: true
        body:
            type: "object"
            required: true

• here is the payload in raw-pp.json

{
  "correlationDetails": {
    "sourceid": "21494e29-5706-11ea-8197-00000a1a06b5"
  },
  "createdTime": "2020-03-18T05:47:55.558Z",
  "description": "Source ID: 21494e29-5706-11ea-8197-00000a1a06b5",
  "displayId": "#0290-69-0",
  "eventsURL": "https://example.com/api/query/v1/066d9480-68dc-11ea-b90a-9e92de0ab0d8/events",
  "id": "066d9480-68dc-11ea-b90a-9e92de0ab0d8",
  "incidentURL": "https://example.com/api/query/v1/066d9480-68dc-11ea-b90a-9e92de0ab0d8",
  "lastChanged": "2020-03-18T05:49:59.861Z",
  "notificationState": "Notification",
  "owner": "-",
  "priority": 5,
  "state": "closed",
  "summary": "Source ID: 21494e29-5706-11ea-8197-00000a1a06b5",
  "team": "-",
  "timelineURL": "https://example.com/api/query/v1/066d9480-68dc-11ea-b90a-9e92de0ab0d8/timeline"
}

• then generate workload to the webhook

while true
do
  curl -k -X POST \
  https://mystackstorm.com/api/v1/webhooks/cem-webhook \
 -H "X-Auth-Token: $token" \
 -H "Content-Type: application/json" \
 --data '@raw-pp.json'
done

Expected Results

What did you expect to happen when running the steps above?

Actual Results

What happened? What output did you get?

• It takes several seconds for StackStorm to accomplish such a simple action.

st2 execution list --action mcm_chatops.cem --last 10
+--------------------------+-----------------+--------------+------------------------+----------------------+------------------+
| id                       | action.ref      | context.user | status                 | start_timestamp      | end_timestamp    |
+--------------------------+-----------------+--------------+------------------------+----------------------+------------------+
| 5e71db35b180db753e45c36b | mcm_chatops.cem | root         | succeeded (3s elapsed) | Wed, 18 Mar 2020     | Wed, 18 Mar 2020 |
|                          |                 |              |                        | 08:26:28 UTC         | 08:26:31 UTC     |
| 5e71db35b180db753e45c36c | mcm_chatops.cem | root         | succeeded (4s elapsed) | Wed, 18 Mar 2020     | Wed, 18 Mar 2020 |
|                          |                 |              |                        | 08:26:28 UTC         | 08:26:32 UTC     |
| 5e71db35b180db753e45c36f | mcm_chatops.cem | root         | running (4s elapsed)   | Wed, 18 Mar 2020     |                  |
|                          |                 |              |                        | 08:26:28 UTC         |                  |
| 5e71db35b180db753e45c374 | mcm_chatops.cem | root         | running (3s elapsed)   | Wed, 18 Mar 2020     |                  |
|                          |                 |              |                        | 08:26:29 UTC         |                  |
| 5e71db36b180db753e45c378 | mcm_chatops.cem | root         | running (3s elapsed)   | Wed, 18 Mar 2020     |                  |
|                          |                 |              |                        | 08:26:29 UTC         |                  |
| 5e71db36b180db753e45c37c | mcm_chatops.cem | root         | running (3s elapsed)   | Wed, 18 Mar 2020     |                  |
|                          |                 |              |                        | 08:26:29 UTC         |                  |
| 5e71db36b180db753e45c382 | mcm_chatops.cem | root         | running (2s elapsed)   | Wed, 18 Mar 2020     |                  |
|                          |                 |              |                        | 08:26:30 UTC         |                  |
| 5e71db37b180db753e45c385 | mcm_chatops.cem | root         | scheduled              | Wed, 18 Mar 2020     |                  |
|                          |                 |              |                        | 08:26:30 UTC         |                  |
| 5e71db37b180db753e45c391 | mcm_chatops.cem | root         | requested              | Wed, 18 Mar 2020     |                  |
|                          |                 |              |                        | 08:26:31 UTC         |                  |
| 5e71db38b180db753e45c393 | mcm_chatops.cem | root         | requested              | Wed, 18 Mar 2020     |                  |
|                          |                 |              |                        | 08:26:31 UTC         |                  |
+--------------------------+-----------------+--------------+------------------------+----------------------+------------------+
+------------------------------------------------------------------------------------------------------------------------------+
| Note: Only first 10 action executions are displayed. Use -n/--last flag for more results.                                    |
+------------------------------------------------------------------------------------------------------------------------------+

• and CPU usage are kept exhausted on 8 CPUs

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

Here is my /etc/st2/st2.conf

# System-wide configuration

[api]
# Host and port to bind the API server.
host = 127.0.0.1
port = 9101
logging = /etc/st2/logging.api.gunicorn.conf
mask_secrets = True
# allow_origin is required for handling CORS in st2 web UI.
# allow_origin = http://myhost1.example.com:3000,http://myhost2.example.com:3000

[stream]
logging = /etc/st2/logging.stream.gunicorn.conf

[sensorcontainer]
logging = /etc/st2/logging.sensorcontainer.conf

[rulesengine]
logging = /etc/st2/logging.rulesengine.conf

[actionrunner]
logging = /etc/st2/logging.actionrunner.conf
virtualenv_opts =
workflows_pool_size = 8
actions_pool_size = 8

[resultstracker]
logging = /etc/st2/logging.resultstracker.conf

[notifier]
logging = /etc/st2/logging.notifier.conf

[exporter]
logging = /etc/st2/logging.exporter.conf

[garbagecollector]
logging = /etc/st2/logging.garbagecollector.conf

[timersengine]
logging = /etc/st2/logging.timersengine.conf

[auth]
#host = 127.0.0.1
host = mystackstorm.com
port = 9100
use_ssl = True
debug = False
enable = True
logging = /etc/st2/logging.auth.gunicorn.conf
# Note: Settings below are only used in "standalone" mode
backend = flat_file
backend_kwargs = {"file_path": "/etc/st2/htpasswd"}
#backend = pam
#backend_kwargs = {"service": "login"}
cert = /root/StackStorm/cacert/integrationca.crt
key =  /root/StackStorm/cacert/integrationca.key

# Base URL to the API endpoint excluding the version (e.g. http://myhost.net:9101/)
api_url = https://mystackstorm.com/api/

[system]
base_path = /opt/stackstorm

[webui]
# webui_base_url = https://mywebhost.domain
webui_base_url=https://mystackstorm.com

[syslog]
host = 127.0.0.1
port = 514
facility = local7
protocol = udp

[log]
excludes = requests,paramiko
redirect_stderr = False
mask_secrets = True

[system_user]
user = root
ssh_key_file = /home/stanley/.ssh/id_rsa

[messaging]
url = amqp://guest:guest@127.0.0.1:5672/

[ssh_runner]
remote_dir = /tmp

[workflow_engine]
logging = /etc/st2/logging.workflowengine.conf

[scheduler]
# The maximum number of attempts that the scheduler retries on error.
retry_max_attempt = 10
# Location of the logging configuration file.
logging = /etc/st2/logging.scheduler.conf
# How long (in seconds) to sleep between each action scheduler main loop run interval.
sleep_interval = 0.1
# The size of the pool used by the scheduler for scheduling executions.
pool_size = 10
# The number of milliseconds to wait in between retries.
retry_wait_msec = 3000
# How often (in seconds) to look for zombie execution requests before rescheduling them.
gc_interval = 10


[keyvalue]
encryption_key_path = /etc/st2/keys/datastore_key.json

Start by reproducing the webhook workload with the provided curl loop and inspect the rulesengine, actionrunner, and scheduler behavior alongside st2 execution list. Compare execution latency, queue growth, and CPU usage under load; done means the webhook actions are processed without the reported multi-second backlog and exhausted CPU.