DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect [LUCENE-3022]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
So I reviewed the sourcecode and found the problem:
[code]
protected void decomposeInternal(final Token token) {
// Only words longer than minWordSize get processed
if (token.length() <this.minWordSize) {
return;
}
char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
for (int i=0;i
Contributor guide
Assessment
This issue has not been assessed yet.