apache / apache/lucene

New KoreanDecomposeFilter for KoreanAnalyzer(Nori) [LUCENE-8553]

Open
#9,599 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This is a patch for KoreanDecomposeFilter.

This filter can be used to decompose Hangul.
(ex) 한글 -> ㅎㄱ or ㅎㅏㄴㄱㅡㄹ)

Hangul input is very unique.

If you want to type apple in English,
   you can type it in the order a -> p -> p -> l -> e.

However, if you want to input "Hangul" in Hangul,
   you have to type it in the order of ㅎ -> ㅏ -> ㄴ -> ㄱ -> ㅡ -> ㄹ.
   (Because of the keyboard shape)

This means that spell check with existing full Hangul can be less accurate.

 

The structure of Hangul consists of elements such as **"Choseong"**, **"Jungseong"**, and **"Jongseong"**.

These three elements are called **"Jamo"**.

If you have the Korean word "된장찌개" (that means Soybean Paste Stew)
**"Choseong"** means "ㄷ, ㅈ, ㅉ, ㄱ",
**"Jungseong"** means "ㅚ, ㅏ, ㅣ, ㅐ",
**"Jongseong"** means "ㄴ, ㅇ".

The reason for Jamo separation is explained above. (spell check)

Also, the reason we need "Choseong Filter" is because many Koreans use **"Choseong Search"** (especially in mobile environment).
If you want to search for "된장찌개" you need 10 typing, which is quite a lot.
For that reason, I think it would be useful to provide a filter that can be searched by "ㄷㅈㅉㄱ".

Hangul also has **dual chars**, such as
"ㄲ, ㄸ, ㅁ, ㅃ, ㅉ, ㅚ (ㅗ + ㅣ), ㅢ (ㅡ + ㅣ), ...".

For such reasons,
KoreanDecompose offers **5 options**,

ex) **된장찌개** => [된장], [찌개]

**1) ORIGIN**
[된장], [찌개]

**2) SINGLECHOSEONG**
[ㄷㅈ], [ㅉㄱ]

**3) DUALCHOSEONG**
[ㄷㅈ], [ㅈㅈㄱ]

**4) SINGLEJAMO**
[ㄷㅚㄴㅈㅏㅇ], [ㅉㅣㄱㅐ]

**5) DUALJAMO**
[ㄷㅗㅣㄴㅈㅏㅇ], [ㅈㅈㅣㄱㅐ]

---
Migrated from [LUCENE-8553](https://issues.apache.org/jira/browse/LUCENE-8553) by Namgyu Kim (@danmuzi), updated Nov 02 2018
Attachments: [LUCENE-8553.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8553/LUCENE-8553.patch)

Contributor guide

Open the contributing guide

Research direction

Start with the attached LUCENE-8553.patch and the KoreanAnalyzer (Nori) entry point to understand the proposed filter integration. Check the five decomposition options against the Hangul examples in the issue; done means the filter supports the described origin, choseong, and jamo outputs with the stated dual-character behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.