Experiment: Increase Solr Caches in Production

As I’ve been recreating the solr database and testing one thing seems to come up quite often that should be able to improve our system and that is tweaking the cache config. Unfortunately, this is nearly impossible to do in a meaningful way without production-level traffic, but based on the AI running around and making some estimates, it thinks that these would be relatively good cash numbers, and I think it's worth giving a shot.

I suppose since we have two solr read replicas we can run them each with diff caches and see if they act differently?

CC: @cdrini

## Purpose

* Test larger Solr caches on production nodes.
* Goal: Increase throughput and reduce tail latency under diverse load.

## Background

* Current cache sizes are small: `filterCache 512`, `queryResultCache 512`, `documentCache 512`, `perSegFilter 10`.
* Refer to `conf/solr/conf/solrconfig.xml:351-383`.
* Load test with diverse corpus shows many cache misses. Refer to `solr_performance_findings.md:128-135`.
* `perSegFilter` at size `10` has `2.48M` lookups with `2` misses. It blocks block-join queries.

## Change

* Change four values in `solrconfig.xml:351-383`:
* `filterCache` size `512` to `2048`
* `queryResultCache` size `512` to `8192`
* `documentCache` size `512` to `4096`
* `perSegFilter` size `10` to `256`
* Keep `autowarmCount 128` for `filterCache` and `queryResultCache`.
* No code change. No reindex needed.

## Why This Value

* Test with `20,000` diverse queries (larger than cache) shows:
* Baseline `512/512/512/10`: `267` qps, p99 `590` ms
* New `2048/8192/4096/256`: `454` qps (+70%), p99 `299` ms (-49%)
* `full_html` profile: `143` to `177` qps (+24%)
* Refer to `solr_performance_findings.md:128-132`.
* RAM on test host stayed at `7.2` GB with heap `4` GB. Safe.
* First test with `5,000` corpus gave false `2,500` qps because all queries fit in `8,192` slots. Server p50 was `0` ms. Test with `20,000` corpus is valid.

## How To Experiment

1. Deploy new `solrconfig.xml` to one Solr node first (e.g. `ol-solr2`), or to all nodes if single-node deploy is not possible.
2. Reload core: `curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=openlibrary"`.
3. Wait `15` seconds for warm-up to finish.
4. Watch for `24` to `48` hours:
* `solr/admin/metrics?core=openlibrary` — `filterCache` and `queryResultCache` hit/miss, `indexsearcher_cache_ram_used_bytes`
* `/select` QTime p95 and p99 from Grafana `ol.label` buckets
* Error rate and heap use (`docker stats`)
5. Compare to same hours in prior week.

## Success Criteria

* p99 down, throughput up, hit ratio up.
* RAM stays below heap limit.
* No increase in errors.

## Risks

* More RAM use. Measured increase is small (`~1` GB). Watch heap.
* If RAM is high, reduce `queryResultCache` to `4096`.

## Rollback

* Change four numbers back to `512/512/512/10` and reload. No data change. Downtime is only reload time (`1` to `2` seconds).

Start with conf/solr/conf/solrconfig.xml:351-383 and solr_performance_findings.md:128-135 to review the proposed cache values and supporting measurements. Deploy to a Solr node, reload the openlibrary core with the provided curl command, and monitor Solr cache metrics, Grafana /select latency, errors, and docker stats for 24-48 hours. Done means improved p99 and throughput without exceeding the heap or increasing errors; revert if RAM is high.

Written by the indexing model from the issue text.