apache / apache/lucene

German/French stemmers fail for common forms maux, gegrüßt, grüßend, schlummert [LUCENE-9410]

Open
#10,450 1 comment 0 reactions 0 assignees View on GitHub
affects-version:8.5 legacy-jira-label:french legacy-jira-label:german legacy-jira-label:stemmer legacy-jira-label:stemming legacy-jira-priority:Critical module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I'm using Lucene via Elasticsearch 7.7.1. German and French stemmers (either via the Snowball analyzer, or the "light" or "heavy" stemming analyzers) are failing to understand some common forms:

French:
- "maux" (plural) should match "mal" (singular) but instead "maux" is unchanged

German:
- "schlummert" should match "schlummern" (infinitive) but instead is unchanged
- "grüßend" should match "grüßen" (infinitive) but instead yields "grussend"
- "gegrüßt" should match "grüßen" (infinitive) but instead yields "gegrusst"

The Elasticsearch folks [said](https://discuss.elastic.co/t/better-french-and-german-stemming/236283) I should file a bug with Lucene.

---
Migrated from [LUCENE-9410](https://issues.apache.org/jira/browse/LUCENE-9410) by Ben Kazez, updated Oct 20 2020
Environment:
```
Elasticsearch 7.7.1 running on cloud.elastic.co
```

Pull requests: https://github.com/apache/lucene-solr/pull/1983

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported forms using Lucene's German and French stemmers through the Snowball, light, and heavy analyzers, with Elasticsearch 7.7.1 as the reported environment. Review the relevant stemmer entry points and pull request 1983. Done means the listed forms produce stems that match their expected singular or infinitive forms.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.