German/French stemmers fail for common forms maux, gegrüßt, grüßend, schlummert [LUCENE-9410]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I'm using Lucene via Elasticsearch 7.7.1. German and French stemmers (either via the Snowball analyzer, or the "light" or "heavy" stemming analyzers) are failing to understand some common forms:
French:
- "maux" (plural) should match "mal" (singular) but instead "maux" is unchanged
German:
- "schlummert" should match "schlummern" (infinitive) but instead is unchanged
- "grüßend" should match "grüßen" (infinitive) but instead yields "grussend"
- "gegrüßt" should match "grüßen" (infinitive) but instead yields "gegrusst"
The Elasticsearch folks [said](https://discuss.elastic.co/t/better-french-and-german-stemming/236283) I should file a bug with Lucene.
---
Migrated from [LUCENE-9410](https://issues.apache.org/jira/browse/LUCENE-9410) by Ben Kazez, updated Oct 20 2020
Environment:
```
Elasticsearch 7.7.1 running on cloud.elastic.co
```
Pull requests: https://github.com/apache/lucene-solr/pull/1983
Contributor guide
Research direction
Reproduce the reported forms using Lucene's German and French stemmers through the Snowball, light, and heavy analyzers, with Elasticsearch 7.7.1 as the reported environment. Review the relevant stemmer entry points and pull request 1983. Done means the listed forms produce stems that match their expected singular or infinitive forms.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100