Add decompose compound Japanese Katakana token capability to Kuromoji [LUCENE-3921]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Japanese morphological analyzer, Kuromoji doesn't have a capability to decompose every Japanese Katakana compound tokens to sub-tokens. It seems that some Katakana tokens can be decomposed, but it cannot be applied every Katakana compound tokens. For instance, "トートバッグ(tote bag)" and "ショルダーバッグ" don't decompose into "トート バッグ" and "ショルダー バッグ" although the IPA dictionary has "バッグ" in its entry. I would like to apply the decompose feature to every Katakana tokens if the sub-tokens are in the dictionary or add the capability to force apply the decompose feature to every Katakana tokens.
---
Migrated from [LUCENE-3921](https://issues.apache.org/jira/browse/LUCENE-3921) by Kazuaki Hiraga (@hkazuakey), updated Oct 07 2012
Environment:
```
Cent OS 5, IPA Dictionary, Run with "Search mdoe"
```
Contributor guide
Research direction
Start with the Kuromoji Japanese morphological analyzer and reproduce the reported behavior in Search mode using the IPA Dictionary, focusing on the Katakana compounds "トートバッグ" and "ショルダーバッグ". Determine how existing decomposition handles dictionary sub-tokens and define completion as decomposing applicable compounds into forms such as "トート バッグ" and "ショルダー バッグ", or providing the requested forced-decomposition capability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100