Handle punctuation characters in KoreanTokenizer [LUCENE-8977]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
As we discussed on #10009, KoreanTokenizer always divides into one and the others now when there are continuous punctuation marks.
(사이즈.... => [사이즈] [.] [...])
But KoreanTokenizer doesn't divide when first character is punctuation.
(...사이즈 => [...] [사이즈])
It looks like the result from the viterbi path, but users can think weird about the following case:
("사이즈" means "size" in Korean)
|Case #1|Case #2|
|-|-|
|Input : "...사이즈..."|Input : "...4......4사이즈"|
|Result : [...] [사이즈] [.] [..]|Result : [...] [4] [.] [.....] [4] [사이즈]|
From what I checked, Nori has a punctuation characters(like . ,) in the dictionary but Kuromoji is not.
("サイズ" means "size" in Japanese)
|Case #1|Case #2|
|-|-|
|Input : "...サイズ..."|Input : "...4......4サイズ"|
|Result : [...] [サイズ] [...]|Result : [...] [4] [......] [4] [サイズ]|
There are some ways to resolve it like hard-coding for punctuation but it seems not good.
So I think we need to discuss it.
---
Migrated from [LUCENE-8977](https://issues.apache.org/jira/browse/LUCENE-8977) by Namgyu Kim (@danmuzi), updated Sep 18 2019
Contributor guide
Research direction
The issue names KoreanTokenizer but no source file or test. Start by reproducing the listed punctuation cases and inspecting the current tokenizer behavior, then compare Nori and Kuromoji dictionary handling. Done is not defined until the desired tokenization and implementation approach are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100