Improve the leaf search scheduler logic.

Current state

When running several requests concurrently, concurrency is controlled by the following rudimentary tools.
A semaphore makes sure we run at most N (N=100 by default) concurrent leaf search.
The goal of this semaphore is mostly to make sure we do not run out of RAM, and that we do not end up dilluting bandwidth in such a way that each download takes more time than if they were running more sequentially.

If N is too low, we cannot exhaust the network bandwidth.
If N is too large, we end up overusing memory linearly. We also may end up hurting the distribution of download time. In particular, the time to the first CPU will be affected.

The CPU intensive part of the search ( the actual search execution) is then running in a thread pool sized against the number of CPU cores.

Room for improvement

When running concurrent requests made of the same number of tasks (assumed of same size), it is intuitively to pick an order between the two, and run the tasks of request 1 in priority before the tasks of request 2.

That way

• request 1 will return earlier.
• request 2 will return as fast as if we were running tasks in random order.

In addition, when running a short task and a long task, short tasks should run in priority.
One way to get convince of that is to realize that if the system is running at capacity,
minimizing the average latency is equivalent to minimize the number of people waiting for a response.
Running short requests first is a way to decrease the number of people waiting for a response.

Start by locating the leaf search scheduler, its semaphore limiting concurrent searches, and the CPU-sized thread pool described in the issue. Read how requests and leaf tasks are queued, then define and validate a scheduling approach that prioritizes tasks from earlier requests and shorter tasks while preserving concurrency and resource limits.