apache / apache/lucene

HyphenationCompoundWordTokenFilter does not work correctly with the german word Brustamputation [LUCENE-3047]

Open
#4,120 0 comments 0 reactions 0 assignees View on GitHub
affects-version:3.1 legacy-jira-priority:Minor module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Following Test fails:

`@Test`
public void testBrustamputation()
throws IOException {
Analyzer compoundAnalyzer = new Analyzer()
{
`@Override`
public TokenStream tokenStream( String fieldName, Reader reader ) {
InputStream in = this.getClass().getResourceAsStream( "/de_DR.xml" );

final InputSource inputSource = new InputSource( in );
inputSource.setEncoding( "iso-8859-1" );
HyphenationTree hyphenator = null;
try {
hyphenator = HyphenationCompoundWordTokenFilter.getHyphenationTree( inputSource );
} catch ( Exception ex ) {
Assert.fail( "", ex);
}
HashSet dict = new HashSet( Arrays.asList( new String[]{"brust", "amputation"} ) );
return new HyphenationCompoundWordTokenFilter( Version.LUCENE_31, new WhitespaceTokenizer( Version.LUCENE_31, reader ), hyphenator,
dict, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
4, CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, false );
}
};
TokenStream tokenStream = compoundAnalyzer.tokenStream( "Kurztext", new StringReader( "brustamputation" ) );
CharTermAttribute t = tokenStream.addAttribute( CharTermAttribute.class );
Set<String> tokenSet = new HashSet<String>();
while ( tokenStream.incrementToken() ) {
tokenSet.add( t.toString() );
System.out.println( t );
}
Assert.assertTrue( tokenSet.contains( "brust" ), "brust" );
Assert.assertTrue( tokenSet.contains( "brustamputation" ), "brustamputation" );
Assert.assertTrue( tokenSet.contains( "amputation" ), "amputation" );

}

---
Migrated from [LUCENE-3047](https://issues.apache.org/jira/browse/LUCENE-3047) by Lars Feistner, updated May 16 2011
Environment:
```
Linux 2.6.32-31-generic
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
```

Contributor guide

Open the contributing guide

Research direction

Start by locating and running the testBrustamputation test with the de_DR.xml resource, then inspect HyphenationCompoundWordTokenFilter and its compound-word handling. Done means the token stream for "brustamputation" contains brust, brustamputation, and amputation as asserted by the test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.