apache / apache/lucene

DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect [LUCENE-3022]

Open
#4,096 8 comments 0 reactions 1 assignee Claimed by @rmuir View on GitHub
affects-version:2.9.4 affects-version:3.1 legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-label:dead legacy-jira-priority:Minor module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
So I reviewed the sourcecode and found the problem:
[code]
protected void decomposeInternal(final Token token) {
// Only words longer than minWordSize get processed
if (token.length() <this.minWordSize) {
return;
}

char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());

for (int i=0;i

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.