apache / apache/lucene

ThaiAnalyzer fail to tokenize word. [LUCENE-4253]

Open
#5,323 9 comments 0 reactions 0 assignees View on GitHub
affects-version:Realtime Branch legacy-jira-priority:Major module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Method
protected TokenStreamComponents createComponents(String,Reader)

return a component that unable to tokenize Thai word.
The current return statement is:
return new TokenStreamComponents(source, new StopFilter(matchVersion, result, stopwords));

My experiment is change the return statement to:
return new TokenStreamComponents(source, result);

It give me a correct result.

---
Migrated from [LUCENE-4253](https://issues.apache.org/jira/browse/LUCENE-4253) by Nattapong Sirilappanich, updated Jul 30 2012
Environment:
```
Windows 7 SP1.
Java 1.7.0-b147
```

Contributor guide

Open the contributing guide

Research direction

Start at ThaiAnalyzer's createComponents(String, Reader) method and inspect how StopFilter affects the returned TokenStreamComponents. Reproduce the reported Thai-word tokenization result, then confirm the intended stopword behavior and add or update a regression test so the expected tokens are preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.