DictionaryCompoundWordTokenFilter should respect minSubwordSize also for fragments [LUCENE-6809]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
`DictionaryCompoundWordTokenFilter` is very useful for building German search indices. However this can also lead to ambiguities as words might be extracted which have a completely different meaning. Most often this happens if the remaining parts of the word are too small.
Example: `schwein` (German for pig) contains `wein` (German for wine). Even if `minSubwordSize` is set to `4`, `wein` gets extracted though `sch` is shorter than 4 characters.
We could solve this by requiring all word parts to be part of the dictionary (at most 3), but this creates problems with compound words of more than three nouns.
Therefore we have built an alternate solution where `minSubwordSize` is also applied to the rest of the fragments. We have tested this in several (large) customer indices and it is working much better than before.
---
Migrated from [LUCENE-6809](https://issues.apache.org/jira/browse/LUCENE-6809) by Christian Winkler, updated May 09 2016
Attachments: [LUCENE-6809.diff](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6809/LUCENE-6809.diff)
Contributor guide
Research direction
Start by locating DictionaryCompoundWordTokenFilter and the code that applies minSubwordSize to compound-word fragments. Reproduce the schwein example with minSubwordSize set to 4, then verify that fragments are emitted only when the remaining parts also meet the minimum size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100