apache / apache/lucene

DictionaryCompoundWordTokenFilter should respect minSubwordSize also for fragments [LUCENE-6809]

Open
#7,867 1 comment 0 reactions 0 assignees View on GitHub
affects-version:5.3 affects-version:6.0 legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

`DictionaryCompoundWordTokenFilter` is very useful for building German search indices. However this can also lead to ambiguities as words might be extracted which have a completely different meaning. Most often this happens if the remaining parts of the word are too small.

Example: `schwein` (German for pig) contains `wein` (German for wine). Even if `minSubwordSize` is set to `4`, `wein` gets extracted though `sch` is shorter than 4 characters.

We could solve this by requiring all word parts to be part of the dictionary (at most 3), but this creates problems with compound words of more than three nouns.

Therefore we have built an alternate solution where `minSubwordSize` is also applied to the rest of the fragments. We have tested this in several (large) customer indices and it is working much better than before.

---
Migrated from [LUCENE-6809](https://issues.apache.org/jira/browse/LUCENE-6809) by Christian Winkler, updated May 09 2016
Attachments: [LUCENE-6809.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6809/LUCENE-6809.diff)

Contributor guide

Open the contributing guide

Research direction

Start by locating DictionaryCompoundWordTokenFilter and the code that applies minSubwordSize to compound-word fragments. Reproduce the schwein example with minSubwordSize set to 4, then verify that fragments are emitted only when the remaining parts also meet the minimum size.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.