internetarchive / internetarchive/openlibrary

Tweak solr cache configs

Open
#11,472 8 comments 0 reactions 0 assignees View on GitHub
Lead: @cdrini Module: Solr Priority: 2 Theme: Performance
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

We currently have two caches that have very high hit rates, the `filterCache` and the `perSegFilter` cache

`filterCache` is I believe used whenever with use the `fq` parameter. We use this a lot of things like filtered author queries, reading log queries, etc.

Image

`perSegFilter` is apparently "custom cache currently used by block join". I believe we use block join indirectly for our edition-aware queries, which power huge chunks of the site.

Image

Increasing the sizes of these caches might improve solr performance.

### Stakeholders

@scottbarnes @tfmorris

Contributor guide

Open the contributing guide

Research direction

Start by locating the Solr configuration entries for the filterCache and perSegFilter caches, then review the linked cache hit-rate evidence and their current sizes. Compare potential size increases against observed Solr performance and define completion as a validated configuration change that improves performance without unacceptable resource use.

Written by the indexing model from the issue text.

Assessment

Domain
backend, performance, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.