apache / apache/lucene

EnglishMinimalStemmer does not implement s-stemmer paper correctly? [LUCENE-8876]

Open
#9,919 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The EnglishMinimalStemmer fails to stem ees suffixes like bees, trees and employees.

The [original paper\|[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.9828&rep=rep1&type=pdf]] has this table of rules:

![https://user-images.githubusercontent.com/170925/59616454-5dc7d580-911c-11e9-80b0-c7a59458c5a7.png](https://user-images.githubusercontent.com/170925/59616454-5dc7d580-911c-11e9-80b0-c7a59458c5a7.png)

The notes accompanying the table state :
> "the first applicable rule encountered is the only one used"

 

For the `ees` and `oes` suffixes I think EnglishMinimalStemmer misinterpreted the rule logic and consequently `bees != bee` and `tomatoes != tomato`. The `oes` and `ees` suffixes are left intact.

"The first applicable rule" for `ees` could be interpreted as rule 2 or 3 in the table depending on if you take `applicable` to mean "the THEN part of the rule has fired" or just that the suffix was referenced in the rule. EnglishMinimalStemmer has assumed the latter and I think it should be the former. We should fall through into rule 3 for `ees` and `oes` (remove any trailing S). That's certainly the conclusion I came to independently testing on real data.

There are some additional changes I'd like to see in a plural stemmer but I won't list them here - the focus should be making the code here match the original paper it references.

---
Migrated from [LUCENE-8876](https://issues.apache.org/jira/browse/LUCENE-8876) by Mark Harwood (@markharwood), updated Jul 01 2019

Contributor guide

Open the contributing guide

Research direction

Start by reading EnglishMinimalStemmer and comparing its suffix-rule order with the linked s-stemmer paper. Check the reported words—bees, trees, employees, and tomatoes—against the paper's first-applicable-rule interpretation. Done means the implementation matches the paper and the reported plural forms stem as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.