internetarchive / internetarchive/openlibrary
Solr applies English stemming to all text, even non-English text
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 138
Description
Currently, Open Library search applies the Porter English stemmer to all text, including non-English words, which can cause issues such as incorrect matching of words and noisy search results.
### Proposal & Constraints
To resolve this issue, Open Library search needs to be made language-aware for stemming. Stemming needs to be done per-language, either using Solr's language detection or data from the MARC record which indicates the language of cataloging.
Open questions:
- At index time, we can use the language on the edition, but what about at if there is no language? Use solr auto-detection options?
- Here are index-time lang detection: https://solr.apache.org/guide/8_11/detecting-languages-during-indexing.html
- We'd need to apply the same stemming rule at query time. This is difficult because we'd need to make assumptions about what language the user's query is in. Unless solr has an option to detect that? But haven't seen any such options in the docs
### Related files
- https://github.com/internetarchive/openlibrary/blob/ec3183a16c6678ed575eebe0bc51734887d4e72b/conf/solr/conf/managed-schema#L599-L625
### Stakeholders
@cdrini @tfmorris
Contributor guide
Research direction
Start with conf/solr/conf/managed-schema around lines 599-625 and the Solr language-detection indexing documentation linked in the issue. Determine how edition or MARC language data and language-less records should be handled at index time, and how query-time language selection should work. Done means non-English text is not processed by the English stemmer and indexing and querying use compatible language-aware rules.
Written by the indexing model from the issue text.
Assessment
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100