internetarchive / internetarchive/openlibrary

Subjects not being indexed into Solr

Open
#1,896 15 comments 0 reactions 0 assignees View on GitHub
Lead: @cdrini Module: Solr Priority: 3 Type: Bug
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

## Description
As far as I can tell, the results for [/search/subjects](https://openlibrary.org/search/subjects?q=guide) are never up-to-date because 1) subjects are never indexed into solr, and 2) even if they were, they always error.

## Evidence

1. Subjects are never indexed into solr
- On local, querying subjects never yields results
- On local, [querying solr for subjects](http://192.168.99.100:8983/solr/select/?q=type%3Asubject&version=2.2&start=0&rows=10&indent=on) yields no results.
2. Subject solr updates always error
- On local, forcing an update to subjects causes an error because `pass` places `None`s in the request array ( `PYTHONPATH=$(pwd) python openlibrary/solr/update_work.py /subjects/voyages_around_the_world /subjects/description_and_travel -s http://0.0.0.0/ -c conf/openlibrary.yml --data-provider=legacy` )

## Technical notes

### 1: Subjects are never indexed into solr

`new-solr-updater` specifically only passes `books`, `authors`, and `works` to `update_works`:

https://github.com/internetarchive/openlibrary/blob/cf14f6a0cdf1623b9bf6bd7d07b43cf01191e8e1/scripts/new-solr-updater.py#L172

And there's no point in `update_works` where a work's subjects are extracted for indexing.

### 2: Subject solr updates always error

The section in `update_works` that deals with updating subjects:

https://github.com/internetarchive/openlibrary/blob/5fe3e192228c2c392b700c1e7ba3e4a2ca910321/openlibrary/solr/update_work.py#L1100-L1107

But the `SolrRequestSet` does nothing when an `add` operation is serialized; this causes the whole chunk of updates to error (so neither the delete or the update are performed, I believe).

https://github.com/internetarchive/openlibrary/blob/5fe3e192228c2c392b700c1e7ba3e4a2ca910321/openlibrary/solr/update_work.py#L934-L957

## Expectation
- Whenever a work is re-indexed, its subject should also be re-indexed
- `SolrRequestSet` shouldn't have a `pass` block

## Relationships
- Might block #1843 and #1067; subject search currently only works on prod because it used to work at some point in the past, and all the data is still in solr. But a new index would need a way to re-index subjects so that subjects search works correctly.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/new-solr-updater.py around the update_works call, then inspect openlibrary/solr/update_work.py at the subject-update and SolrRequestSet sections. Reproduce the failure with the subject update command in the issue. Done means reindexing a work also reindexes its subject and SolrRequestSet no longer fails on the subject update request.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.