[BUG] Search thread pool exceeded by security analytics detectors

We are seeing a large number of rejected search threads after enabling a detector for windows logs with many sigma rules enabled. After enabling the detector, we see the search thread queue bump up to 1,000 every couple seconds. Over the last hour we see several million rejected searches (see attchment).

We noticed this while in the Discover page and querying our data. The searches started to fail frequently with

rejected_execution_exception
rejected execution of org.opensearch.common.util.concurrent.TimedRunnable@14a1906d on QueueResizingOpenSearchThreadPoolExecutor[name = os-01-data-01/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 13.4ms, adjustment amount = 50, org.opensearch.common.util.concurrent.QueueResizingOpenSearchThreadPoolExecutor@791a2b55[Running, pool size = 49, active threads = 49, queued tasks = 1000, completed tasks = 13607928]]
Error: Too Many Requests
    at fetch_Fetch.fetchResponse (http://192.168.60.30:5601/6614/bundles/core/core.entry.js:15:177222)
    at async interceptResponse (http://192.168.60.30:5601/6614/bundles/core/core.entry.js:15:172640)
    at async http://192.168.60.30:5601/6614/bundles/core/core.entry.js:15:175120

Steps to reproduce the behavior:

1. Add a detector for windows logs - maping fields, etc.
2. Start the detector and then run the following comand to see the thread queue
   GET _cat/thread_pool?v&s=rejected:desc
3. Watch for the rejected count to start incrementing
4. Watch for search error in the discover screen (you may have to try several several in order to get a search error)
5. If you disable the detector - then the issue seems to go away. Thread queue is more normal, and searches in discover dont seem to fail.

What is the expected behavior?
The analytics detectors should be able to moderate their activity against the cluster so as to not adversely affect other search activity, as well as allow the detection searches to complete without failing (I think that's what the "rejected" is indicating. Not sure if they are retrying or not)

What is your host/environment?

• OS: 7 node cluster on oracle linux 8.8 with ssd local storage
• OpenSearch 2.10 - fresh install
• Plugins - default installed/enabled

Do you have any additional context?
Just before submitting this report - I noticed that two of our data nodes seemed to stop responding. They both heap dumped with a 30gb heap and 96 gb total ram. There is only one windows log index with about 50gb of data. After restarting the nodes. The detector had timed out. So we stopped it and restarted it. Then the search thread problem started up again. I suspect the analytics detector might be causing our nodes to crash as well. The data ingest and searching has been solid for a week without any issue. These symptoms seem to have started today with the detector enabled in security analytics.

Start by reproducing the issue with a Windows-log detector and many Sigma rules enabled, then monitor GET _cat/thread_pool?v&s=rejected:desc while searching in the Discover page. Compare detector-enabled and detector-disabled behavior; done means detector searches no longer exhaust the search queue or cause rejected Discover searches.