apache / apache/lucene

Handle punctuation characters in KoreanTokenizer [LUCENE-8977]

Open
#10,020 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

As we discussed on #10009, KoreanTokenizer always divides into one and the others now when there are continuous punctuation marks.
(사이즈.... => [사이즈] [.] [...])
But KoreanTokenizer doesn't divide when first character is punctuation.
(...사이즈 => [...] [사이즈])

It looks like the result from the viterbi path, but users can think weird about the following case:
("사이즈" means "size" in Korean)

|Case #1|Case #2|
|-|-|
|Input : "...사이즈..."|Input : "...4......4사이즈"|
|Result : [...] [사이즈] [.] [..]|Result : [...] [4] [.] [.....] [4] [사이즈]|

From what I checked, Nori has a punctuation characters(like . ,) in the dictionary but Kuromoji is not.
("サイズ" means "size" in Japanese)

|Case #1|Case #2|
|-|-|
|Input : "...サイズ..."|Input : "...4......4サイズ"|
|Result : [...] [サイズ] [...]|Result : [...] [4] [......] [4] [サイズ]|

There are some ways to resolve it like hard-coding for punctuation but it seems not good.
So I think we need to discuss it.

---
Migrated from [LUCENE-8977](https://issues.apache.org/jira/browse/LUCENE-8977) by Namgyu Kim (@danmuzi), updated Sep 18 2019

Contributor guide

Open the contributing guide

Research direction

The issue names KoreanTokenizer but no source file or test. Start by reproducing the listed punctuation cases and inspecting the current tokenizer behavior, then compare Nori and Kuromoji dictionary handling. Done is not defined until the desired tokenization and implementation approach are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.