New KoreanDecomposeFilter for KoreanAnalyzer(Nori) [LUCENE-8553]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This is a patch for KoreanDecomposeFilter.
This filter can be used to decompose Hangul.
(ex) 한글 -> ㅎㄱ or ㅎㅏㄴㄱㅡㄹ)
Hangul input is very unique.
If you want to type apple in English,
you can type it in the order a -> p -> p -> l -> e.
However, if you want to input "Hangul" in Hangul,
you have to type it in the order of ㅎ -> ㅏ -> ㄴ -> ㄱ -> ㅡ -> ㄹ.
(Because of the keyboard shape)
This means that spell check with existing full Hangul can be less accurate.
The structure of Hangul consists of elements such as **"Choseong"**, **"Jungseong"**, and **"Jongseong"**.
These three elements are called **"Jamo"**.
If you have the Korean word "된장찌개" (that means Soybean Paste Stew)
**"Choseong"** means "ㄷ, ㅈ, ㅉ, ㄱ",
**"Jungseong"** means "ㅚ, ㅏ, ㅣ, ㅐ",
**"Jongseong"** means "ㄴ, ㅇ".
The reason for Jamo separation is explained above. (spell check)
Also, the reason we need "Choseong Filter" is because many Koreans use **"Choseong Search"** (especially in mobile environment).
If you want to search for "된장찌개" you need 10 typing, which is quite a lot.
For that reason, I think it would be useful to provide a filter that can be searched by "ㄷㅈㅉㄱ".
Hangul also has **dual chars**, such as
"ㄲ, ㄸ, ㅁ, ㅃ, ㅉ, ㅚ (ㅗ + ㅣ), ㅢ (ㅡ + ㅣ), ...".
For such reasons,
KoreanDecompose offers **5 options**,
ex) **된장찌개** => [된장], [찌개]
**1) ORIGIN**
[된장], [찌개]
**2) SINGLECHOSEONG**
[ㄷㅈ], [ㅉㄱ]
**3) DUALCHOSEONG**
[ㄷㅈ], [ㅈㅈㄱ]
**4) SINGLEJAMO**
[ㄷㅚㄴㅈㅏㅇ], [ㅉㅣㄱㅐ]
**5) DUALJAMO**
[ㄷㅗㅣㄴㅈㅏㅇ], [ㅈㅈㅣㄱㅐ]
---
Migrated from [LUCENE-8553](https://issues.apache.org/jira/browse/LUCENE-8553) by Namgyu Kim (@danmuzi), updated Nov 02 2018
Attachments: [LUCENE-8553.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-8553/LUCENE-8553.patch)
Contributor guide
Research direction
Start with the attached LUCENE-8553.patch and the KoreanAnalyzer (Nori) entry point to understand the proposed filter integration. Check the five decomposition options against the Hangul examples in the issue; done means the filter supports the described origin, choseong, and jamo outputs with the stated dual-character behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100