st2 performance issues

SUMMARY

I used k8s to deploy st2, and now the entire workflow action is running slowly, I tried to increase the number of replicas to increase the execution speed, but it doesn't seem to help

STACKSTORM VERSION

st2 3.8.0,on Python 3.8.10

OS, environment, install method

Kubernetes

Steps to reproduce the problem

The number of pods for each of my microservices in k8s is as follows

1. st2sensor 4 Pods, sensor only kafka trigger is in use, kafka has 4 partitions, so I used 4 sensor Pods.
2. st2actionrunner 30 Pods
3. st2workflowengine 30 pods
4. st2rulesengine 30 pods
5. st2scheduler 20 pods
6. st2notifier 20 pods
7. st2garbagecollector 1 pods , the garbage collection mechanism is as follows:
   [garbagecollector]
   action_executions_ttl = 3
   action_executions_output_ttl = 3
   trigger_instances_ttl = 3
   traces_ttl = 3
   rule_enforcements_ttl = 3
   workflow_executions_ttl = 3
   task_executions_ttl = 3
   tokens_ttl = 3
8. st2client 1 pods
9. st2auth 2 pods
10. st2api 2 pods
11. st2stream 2 pods

Expected Results

Actual Results

I've looked at mongo's slow queries before, adding composite indexes to increase the query speed, and now that I see some indexes in mongo that are never used, the next step is probably to remove all unused indexes from mongo.
According to the monitoring, it seems that too few plays can be matched. Is it necessary to increase the number of Pods for st2rulesengine? Do you have any good suggestions?

Thanks!

Start with the Kubernetes deployment topology, monitoring data, and MongoDB slow-query and unused-index observations described in the issue. Compare workflow latency and play matching across the listed service replicas, then identify and validate the limiting component. Done means documenting a reproducible bottleneck and a confirmed configuration or code change that improves execution speed.