internetarchive / internetarchive/openlibrary

Semi-Automate Quarterly Solr Re-Index

Open
#10,291 3 comments 0 reactions 0 assignees View on GitHub
Lead: @cdrini Module: Solr Needs: Breakdown Needs: Staff / Internal Priority: 2 Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

### Proposal

As a followup to:
* #10287

Create a hands-free, scheduled task to routinely run solr optimize (and possibly automated monthly re-indexing)

### Justification

Over time, solr performance can get very bad if we don't regularly run optimize. See #10287

The entire site grinds to a halt

### Breakdown

#### Requirements Checklist

* [ ] move the docker build step for `solr_builder` outside jenkins to #10440
* [ ] Update documentation for running `solrbuilder` via jenkins on `ol-solr1`
* [ ] Upload the solr index to Archive.org item `ol-solr-index` (store last and current)
* [ ] Automate (hands-free) the process of moving `ol-solr1` to `ol-solr0` via #5877
* [ ] Setup a jenkins monthly schedule for re-index (and be careful of when jenkins down due to `ol-solr1`)
* If we take jenkins down to test `ol-solr1` against production, we should have a checklist, apart of which makes sure no upcoming re-index is imminently scheduled.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.