ThaiAnalyzer fail to tokenize word. [LUCENE-4253]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Method
protected TokenStreamComponents createComponents(String,Reader)
return a component that unable to tokenize Thai word.
The current return statement is:
return new TokenStreamComponents(source, new StopFilter(matchVersion, result, stopwords));
My experiment is change the return statement to:
return new TokenStreamComponents(source, result);
It give me a correct result.
---
Migrated from [LUCENE-4253](https://issues.apache.org/jira/browse/LUCENE-4253) by Nattapong Sirilappanich, updated Jul 30 2012
Environment:
```
Windows 7 SP1.
Java 1.7.0-b147
```
Contributor guide
Research direction
Start at ThaiAnalyzer's createComponents(String, Reader) method and inspect how StopFilter affects the returned TokenStreamComponents. Reproduce the reported Thai-word tokenization result, then confirm the intended stopword behavior and add or update a regression test so the expected tokens are preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100