Execution rate goes down considerably when the outstanding requests are high

SUMMARY

As a part of scale testing, I have observed that execution rate goes down considerably, when the outstanding requests are high

STACKSTORM VERSION

st2 3.8.0, on Python 3.8.10

OS, environment, install method

Post what OS you are running this on, along with any other relevant information/

Kubernetes HA, installed through helm install (oneline)

Steps to reproduce the problem

Scale up action pods to a certain number. for ex. 17. The default available threads for actions are 17x60

Each action using python runner with a sleep of 5 sec

1000 req, initiated (python threading - http api calls) gets completed in 77sec

Based on above numbers, the test case - [ Send 2500 req with sleep of 250sec, and 500 req with sleep of 250sec ] in loop

Expected Results

Given enough time for executions based on above numbers, there should not be any outstanding requests after each cycle

Actual Results

But there is pending actions, at the end of each cycle(observed in requested state). This count goes higher after few cycles. Then, the execution rate which is at 17x60 goes down to single digits(as the outstanding count goes from 5k to 20k and more)

Observed the same results with a clean mongodb, and size of 5+million docs

st2scheduler does not process requests, and hence actionrunner too. Resources are minimally used when the execution rate goes slower.

Is there a workaround to enforce scheduler to take up more requests ? When more requests are pending, the pods should work harder, but it goes the other way here

Thanks!

Start by reviewing the st2scheduler and actionrunner entry points, then reproduce the Kubernetes scale test with the stated action counts, sleep durations, and outstanding-request growth. Compare scheduler processing and resource use as pending actions increase. Done means identifying the cause of the throughput collapse and verifying that the repeated cycles no longer accumulate pending actions.