internetarchive / internetarchive/openlibrary
Subjects not being indexed into Solr
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
## Description
As far as I can tell, the results for [/search/subjects](https://openlibrary.org/search/subjects?q=guide) are never up-to-date because 1) subjects are never indexed into solr, and 2) even if they were, they always error.
## Evidence
1. Subjects are never indexed into solr
- On local, querying subjects never yields results
- On local, [querying solr for subjects](http://192.168.99.100:8983/solr/select/?q=type%3Asubject&version=2.2&start=0&rows=10&indent=on) yields no results.
2. Subject solr updates always error
- On local, forcing an update to subjects causes an error because `pass` places `None`s in the request array ( `PYTHONPATH=$(pwd) python openlibrary/solr/update_work.py /subjects/voyages_around_the_world /subjects/description_and_travel -s http://0.0.0.0/ -c conf/openlibrary.yml --data-provider=legacy` )
## Technical notes
### 1: Subjects are never indexed into solr
`new-solr-updater` specifically only passes `books`, `authors`, and `works` to `update_works`:
https://github.com/internetarchive/openlibrary/blob/cf14f6a0cdf1623b9bf6bd7d07b43cf01191e8e1/scripts/new-solr-updater.py#L172
And there's no point in `update_works` where a work's subjects are extracted for indexing.
### 2: Subject solr updates always error
The section in `update_works` that deals with updating subjects:
https://github.com/internetarchive/openlibrary/blob/5fe3e192228c2c392b700c1e7ba3e4a2ca910321/openlibrary/solr/update_work.py#L1100-L1107
But the `SolrRequestSet` does nothing when an `add` operation is serialized; this causes the whole chunk of updates to error (so neither the delete or the update are performed, I believe).
https://github.com/internetarchive/openlibrary/blob/5fe3e192228c2c392b700c1e7ba3e4a2ca910321/openlibrary/solr/update_work.py#L934-L957
## Expectation
- Whenever a work is re-indexed, its subject should also be re-indexed
- `SolrRequestSet` shouldn't have a `pass` block
## Relationships
- Might block #1843 and #1067; subject search currently only works on prod because it used to work at some point in the past, and all the data is still in solr. But a new index would need a way to re-index subjects so that subjects search works correctly.
Contributor guide
Research direction
Start with scripts/new-solr-updater.py around the update_works call, then inspect openlibrary/solr/update_work.py at the subject-update and SolrRequestSet sections. Reproduce the failure with the subject update command in the issue. Done means reindexing a work also reindexes its subject and SolrRequestSet no longer fails on the subject update request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100