internetarchive / internetarchive/openlibrary

Enable stopword detection for solr

Open
#5,393 2 comments 0 reactions 1 assignee Claimed by @SomeInternet View on GitHub
Lead: @cdrini Module: Solr Priority: 3 Theme: Search Type: Feature Request
Dominant language
Python
Stars
6.7k
Forks
2k
Avg merge
2d 19h
Merged PRs (30d)
138

Description

Had to remove stopword detection because it wasn't being handled correctly during solr8 query time. See https://github.com/internetarchive/openlibrary/issues/3317#issuecomment-837506502 . Investigate other approaches to see if we can re-enable it. The code change would basically just need to reverse https://github.com/internetarchive/openlibrary/pull/4337/commits/7f6be51f9a0d00024fe790a12493159733fdbbbf .

Steps:

1. Locally, re-enable stopwords ; basically do the reverse of the [commit that disabled it](https://github.com/internetarchive/openlibrary/pull/4337/changes/7f6be51f9a0d00024fe790a12493159733fdbbbf). Note that commit is rather old, so might require some adjustment to apply to the current codebase.
2. Reset the local environment for the solr changes to take place (see https://docs.openlibrary.org/advanced/solr.html#making-changes-to-solr-config ).
3. Rename [this edition and work](http://localhost:8080/works/OL5702375W/The_Mark_of_the_Crown?edition=key%3A/books/OL7361700M) to "The Mark of the crown" so we can test the change (note the two consecutive stopwords, "of" and "the").
4. In the ui search for `The Mark of the Crown` with quotes and without quotes. Does it match? If it does in both cases, then we might be able to re-enable stopwords, and basically make the change permanent!

In the analysis UI, it should match with as well: http://localhost:8983/solr/#/openlibrary/analysis?analysis.fieldvalue=The%20Mark%20of%20the%20Crown&analysis.query=the%20mark%20of%20the%20crown&analysis.fieldname=alternative_title&verbose_output=1 . This UI is a bit tricky to understand, but the issue we were seeing before was that the consecutive stopwords were getting replaced by one token during indexing, but with _two_ tokens during query. In the analyze UI, we should see the same number of tokens on either side at the end.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.