apache / apache/lucene

Provide a version of GermanNormalizationFilter that uses a modified Umlaut mapping

Open
#11,733 0 comments 1 reaction 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

The GermanNormalizationFilter (GermanNormalizationFilterFactory) includes the following mappings: ä/ae -> a, ö/oe -> o, ü/ue -> u and ß -> ss (plus some simple rules when "ue" should not be converted to "u"). This mapping is very uncommon in German. In German, it is common to treat ä and ae, ö and oe, ü and ue, as well as ß and ss as equivalent. The ASCII versions (ae, oe, ue, ss) are used in cases where you cannot use the non-ASCII characters, e.g. when using an English keyboard or when the system doesn't allow these characters. Also the mapping ß -> ss is used when the lowercase letter ß should be uppercased, e.g. in a fully uppercased headline, since there is no uppercase version of ß ("Königsschießen" => "KÖNIGSSCHIESSEN"). Also note that Switzerland does not use ß, but uses ss instead. With the current GermanNormalizationFilter mapping, searching for "Uber" (the company) finds the frequent word "über", which is unexpected, because "u" and "ü" are (normally) not treated as equivalent.

Therefore I would like to see a filter that normalizes German by mapping ä->ae, ö->oe, ü->ue and ß->ss, either by an additional parameter for GermanNormalizationFilter which switches to that mapping (the previous mapping should of course be the default for backward compatibility), or by having a separate filter (GermanNormalizationFilter2?) that implements this mapping.

Using a charfilter instead is not the same, as this is done before the whole filter chain. The new filter should be a drop-in replacement for GermanNormalizationFilter in any desired position in the filter chain. An alternative would be to have something with a configurable mapping like the MappingCharFilterFactory, but usable as a normal filter.

Contributor guide

Open the contributing guide

Research direction

Start by reading GermanNormalizationFilter and GermanNormalizationFilterFactory, then compare their filter-chain behavior with MappingCharFilterFactory. Define how a drop-in normal filter should apply ä→ae, ö→oe, ü→ue, and ß→ss while preserving the requested chain position; done means the new or configurable filter supports that mapping without changing the existing default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.