[Feature Request] [Tiered Caching 3.0] Additional IRC stats

Is your feature request related to a problem? Please describe

As part of tiered caching's 3.0 changes (https://github.com/opensearch-project/OpenSearch/issues/17303) we want to expose some new stats for the IndicesRequestCache.

We can expose a caching latency stat. This could expose issues, for example if the disk cache is performing poorly because of some throttling issue. Latency is generic, so it can be tracked by every ICache implementation, as hits, misses, etc, are right now.

In the TSC we could also use this to tune the took-time thresholds which guard access to the cache. The idea of the policy is to prevent caching queries which can be returned faster than it takes to get a cache hit. We've found 3ms is good but hand-tuning it could be difficult.

Describe the solution you'd like

All ICaches can track latency. We can just add some cumulative total_caching_hit_latency into the StatsHolder objects, and it'd show up in both the total and aggregated sections of the cache stats API response. This can be incremented with the nanoseconds taken by each hit since node restart. We may also want to track the miss latency, including just the time to check that the value isn't there, not the time to compute it. But it's not as obvious that this is useful. If so, we could also track + expose total_caching_miss_latency.

TSC threshold tuning should happen asynchronously. Once a minute, a scheduled thread would check if the average hit latency for the last minute is higher current threshold, or if it's lower than the current threshold minus some extra. If so it can increment/decrement the threshold by some fixed amount, like 2ms. It would do this by changing the cluster setting value. This means we don't have to expose info about this non-generic behavior in the cache stats API, as its changes can already be observed by checking the cluster setting value. This tuning should be enabled/disabled via dynamic cluster setting and if disabled the threshold would revert to its default. This can go as a separate PR from the latency tracking.

Related component

Search:Performance

Describe alternatives you've considered

We could skip the TSC tuning, and just leave it to the user to take some action like switch from TSC -> on heap cache if disk latency grows too high.

Additional context

We originally wanted to report the fraction of requests that are or aren't cacheable as decided in IndicesService.canCache(). This can help users decide whether to enable tiered caching (if the cacheable fraction is low, it may not help much). It can also help them decide whether to change indices.requests.cache.maximum_cacheable_size to allow some size > 0 queries into the cache, and it can help verify that more queries are entering the cache after making that change.

But, this can already be calculated from info in APIs as it's just (cache hits + cache misses) / (total queries), and total queries is in search.query_total from the node stats API.