WordPress / WordPress/openverse

Change text analysis algorithm used to fix stemming issues

Open
#653 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

✨ goal: improvement 💻 aspect: code 🟨 priority: medium 🧱 stack: api
Dominant language
Python
Stars
380
Forks
290
Avg merge
3h 17m
Merged PRs (30d)
8

Description

Problem

We have a whole host of weird results you can get due to stemming issues. Some of these are already explicitly covered in the text analysis configuration, specifically for "animal": https://github.com/WordPress/openverse-api/blob/478f50cdff2834d602a3df8c198f708e010e6685/ingestion_server/ingestion_server/es_mapping.py#L19-L29

Description

I suspect there must be a better way to handle this with ES than manually adding stemmer overrides for things. Why does a search for news for example, prioritise so many results for "New York" or other "new" things that aren't "news" when we certainly have results that match exactly "news"? (It actually seems impossible, at the moment, to construct a query that will actually retrieve things only with the exact form "news").

Right now we use the default "snowball" stemmer (Porter algorithm). We could switch to Porter2 (snowball calls this "English", but ES calls it porter2) and it would at least solve some of these stemming issues: https://snowballstem.org/algorithms/english/stemmer.html

Here's a JavaScript demo of porter2 and you can see news works as expected (it's actually an exceptional condition in porter2 that the original porter algorithm ignores). "animate" and "animal" however still stem down to "anim" so we can't get rid of those stemmer overrides (frustrating!).

Aside from changing the stemmer algorithm though, we could also try to see if there's a way to stop stemming from occuring on exact match searches... or fix exact matches. Either they're not working, or stemming still occurs. If you search for "news" I would expect it to not stem, but currently you'll still get a ton of "New York" type results. I think we should do this in addition to exploring different English stemming algorithms.

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ingestion_server/ingestion_server/es_mapping.py, especially the existing animal stemmer overrides, and reproduce searches for news and exact "news". Compare the current snowball/Porter behavior with porter2 and investigate whether exact-match searches still stem terms. Done means an agreed stemming approach improves these cases and exact searches no longer return stemmed matches.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.