Optimize sharding strategy for query performance

To leverage shard request cache better, we can consider the following changes to our sharding strategy:

1. **More aggressively shard the write index**. Queries to the write index are hard to cache, due to shard refreshes which invalidate the cache. Using a larger (default) number of shards means that we can leverage parallelism better on these indices.

2. To limit the amount of open shards, we can **execute a shrink action on rollover**, which will set an index to read-only and decrease the amount of primary shards. For these read-only shards, we can expect significantly more cache hits, so parallelism is less important here.

3. Instead of rolling over on max_size, we can look at **max_docs and max_age for rollover conditions**:

- If we know the doc/value collection rate of a search request per shard, we can calculate a reasonable threshold for max_docs. We can start thinking in objectives such as “I want a single shard to be able to return data for this search in x seconds”.
- max_age will increase the caching opportunities.. If all documents on a shard match a given range query on a date range field, the shard request query will get rewritten to a match all query, which will result in more cache hits. This will happen more often as indices cover a smaller time range, e.g. we start rolling over every 4 hours instead of 30 days.

See also:
https://github.com/elastic/elasticsearch/issues/74314