EnglishMinimalStemmer does not implement s-stemmer paper correctly? [LUCENE-8876]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
The EnglishMinimalStemmer fails to stem ees suffixes like bees, trees and employees.
The [original paper\|[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.9828&rep=rep1&type=pdf]] has this table of rules:

The notes accompanying the table state :
> "the first applicable rule encountered is the only one used"
For the `ees` and `oes` suffixes I think EnglishMinimalStemmer misinterpreted the rule logic and consequently `bees != bee` and `tomatoes != tomato`. The `oes` and `ees` suffixes are left intact.
"The first applicable rule" for `ees` could be interpreted as rule 2 or 3 in the table depending on if you take `applicable` to mean "the THEN part of the rule has fired" or just that the suffix was referenced in the rule. EnglishMinimalStemmer has assumed the latter and I think it should be the former. We should fall through into rule 3 for `ees` and `oes` (remove any trailing S). That's certainly the conclusion I came to independently testing on real data.
There are some additional changes I'd like to see in a plural stemmer but I won't list them here - the focus should be making the code here match the original paper it references.
---
Migrated from [LUCENE-8876](https://issues.apache.org/jira/browse/LUCENE-8876) by Mark Harwood (@markharwood), updated Jul 01 2019
Contributor guide
Research direction
Start by reading EnglishMinimalStemmer and comparing its suffix-rule order with the linked s-stemmer paper. Check the reported words—bees, trees, employees, and tomatoes—against the paper's first-applicable-rule interpretation. Done means the implementation matches the paper and the reported plural forms stem as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100