CPU never reaching 100%

Quoting @kstaken in #1679

The bad news is that query times are 69-72 seconds with no evidence of caching making any difference. Previously the first query > would be in the 60 second range but subsequent queries dropped to around 30 seconds.
We also tried doubling the cache sizes to confirm we weren't simply churning the cache and it made no difference.
It also never uses much beyond 2 CPUs but maybe that makes sense since it's using async concurrency for the split searches.
Previously it was using 4 CPUs.

For the caching part, I opened the ticket. We use an LRU cache, and your use case is the worst case scenario for an LRU policy.
I noticed that last week. I just created a ticket #1714.

Once the data is downloaded, the computation is executed in a thread pool size. The thread pool is a wrapper around a rayon thread pool. with the same number of threads as the number of CPU.

Here are possible reasons:
a) RAYON_NUM_THREADS env variable was set to 2 for another reason, (unlikely)
b) minio is the bottleneck (unlikely)
c) the number of parallel split is not high enough to get the best throughput off minio (unlikely)
d) network is the bottleneck (unlikely)
e) a straight bug.

c) and d) are the most likely.

I'm adding a bunch of counters that should help identify the problem rapidly.
If you want to test hypothesis c), the idea is just to increase max_num_concurrent_split_searches.

It will increase the RAM usage, and download more split data concurrently.

Start with the Rayon-backed thread pool and the max_num_concurrent_split_searches setting mentioned in the issue. Use the planned counters to investigate whether split concurrency, MinIO, network activity, or a straight bug explains the low CPU usage; completion means identifying the cause and documenting the observed bottleneck.