HyphenationCompoundWordTokenFilter does not work correctly with the german word Brustamputation [LUCENE-3047]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Following Test fails:
`@Test`
public void testBrustamputation()
throws IOException {
Analyzer compoundAnalyzer = new Analyzer()
{
`@Override`
public TokenStream tokenStream( String fieldName, Reader reader ) {
InputStream in = this.getClass().getResourceAsStream( "/de_DR.xml" );
final InputSource inputSource = new InputSource( in );
inputSource.setEncoding( "iso-8859-1" );
HyphenationTree hyphenator = null;
try {
hyphenator = HyphenationCompoundWordTokenFilter.getHyphenationTree( inputSource );
} catch ( Exception ex ) {
Assert.fail( "", ex);
}
HashSet dict = new HashSet( Arrays.asList( new String[]{"brust", "amputation"} ) );
return new HyphenationCompoundWordTokenFilter( Version.LUCENE_31, new WhitespaceTokenizer( Version.LUCENE_31, reader ), hyphenator,
dict, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
4, CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, false );
}
};
TokenStream tokenStream = compoundAnalyzer.tokenStream( "Kurztext", new StringReader( "brustamputation" ) );
CharTermAttribute t = tokenStream.addAttribute( CharTermAttribute.class );
Set<String> tokenSet = new HashSet<String>();
while ( tokenStream.incrementToken() ) {
tokenSet.add( t.toString() );
System.out.println( t );
}
Assert.assertTrue( tokenSet.contains( "brust" ), "brust" );
Assert.assertTrue( tokenSet.contains( "brustamputation" ), "brustamputation" );
Assert.assertTrue( tokenSet.contains( "amputation" ), "amputation" );
}
---
Migrated from [LUCENE-3047](https://issues.apache.org/jira/browse/LUCENE-3047) by Lars Feistner, updated May 16 2011
Environment:
```
Linux 2.6.32-31-generic
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
```
Contributor guide
Research direction
Start by locating and running the testBrustamputation test with the de_DR.xml resource, then inspect HyphenationCompoundWordTokenFilter and its compound-word handling. Done means the token stream for "brustamputation" contains brust, brustamputation, and amputation as asserted by the test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100