[BUG] Do not use the 'search' queue for everything

v 2.11.1

Our cluster was stable on a r5.2xlarge instance, hovering at ~10% CPU usage. Then we enabled windows detectors and even a r5.8xlarge isn't enough.

We were experimenting with detectors. But they essentially brought down our entire instance.
The main issue can be boiled down to the fact that it's all running in the same 'search' queue. The detector UI is backed by 'search', the detectors themselves are backed by 'search' etc.

Why is this the worst idea ever?
Because as the detectors fill up the queue and cause literally millions of searches to be rejected, ~48 Million per hour were observed overnight.
While this is a tuning and scaling issue, it also completely killed ingestion (our spark pipeline kept failing to write to OS and dropped it in our DLQ) and all dashboards no longer work since the UI also uses the 'search' queue.
So it wasn't just detectors that were failing. Everything started to fail. We couldn't even stop the detector because that request kept failing as well.

We have tried tuning the queues, but even a queue size of 100K is still filling up and we're still running into memory issues.

Management wanted us to try to use Detectors as they were hoping we'd no longer have to maintain our own rules engine with Sigma rules. But it can do the job with far less resources on the exact same data set and not affect anything else if it falls behind.

We are no longer moving forward with OS security analytics.

Start by reviewing how detector execution, detector UI requests, ingestion, and dashboards are assigned to the shared 'search' queue. Reproduce the queue saturation and memory-impact behavior described on v2.11.1, then define and test queue isolation so detector load cannot block searches, ingestion, dashboards, or detector-management requests.