Dedicated Root Search Nodes

Is your feature request related to a problem? Please describe

We have a decently large quickwit deployment with a fairly large search volume. Both in terms of amount of data and amount + frequency of search requests issued. As a result our search pool has a lot of disk cache, which can be much more expensive (8TB/node) to scale compared to CPU / RAM. The root search node, the node that receives the initial request and dispatches is also a search node itself. The mixed responsibility can be a little difficult to appropriately allocate resources and scale. We've seen areas of high CPU usage on nodes that we believe are not doing much in terms of executing search queries.

Having dedicates root nodes could also set the stage for being able to search across multiple independent clusters.

Describe the solution you'd like

A way to designate a search node as a root only. These nodes would be responsible only for receiving, parsing, delegating the initial search query request, and aggregating the results from leaf nodes. These nodes should not need a disk based split cache as the work they do wouldn't involve pulling results from splits.

Describe alternatives you've considered

• Simply adding more search nodes
• Making second deployment in the same cluster with a separate kubernetes service in front of it minus a disk cache. All search requests are sent to this secondary service exclusively. This gets us close, but search requests are still delegated to all nodes, including the ones in the secondary deployment. Search requests are generally slower there because there is no disk cache.

No files or tests are named. Start by tracing how search requests are received, delegated to leaf nodes, and aggregated, then inspect how node roles and deployment configuration are represented; done means a root-only role can be designated without requiring a split cache and does not receive leaf search work.