apache / apache/lucene

Add decompose compound Japanese Katakana token capability to Kuromoji [LUCENE-3921]

Open
#4,994 7 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-ALPHA legacy-jira-label:features legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Japanese morphological analyzer, Kuromoji doesn't have a capability to decompose every Japanese Katakana compound tokens to sub-tokens. It seems that some Katakana tokens can be decomposed, but it cannot be applied every Katakana compound tokens. For instance, "トートバッグ(tote bag)" and "ショルダーバッグ" don't decompose into "トート バッグ" and "ショルダー バッグ" although the IPA dictionary has "バッグ" in its entry. I would like to apply the decompose feature to every Katakana tokens if the sub-tokens are in the dictionary or add the capability to force apply the decompose feature to every Katakana tokens.

---
Migrated from [LUCENE-3921](https://issues.apache.org/jira/browse/LUCENE-3921) by Kazuaki Hiraga (@hkazuakey), updated Oct 07 2012
Environment:
```
Cent OS 5, IPA Dictionary, Run with "Search mdoe"
```

Contributor guide

Open the contributing guide

Research direction

Start with the Kuromoji Japanese morphological analyzer and reproduce the reported behavior in Search mode using the IPA Dictionary, focusing on the Katakana compounds "トートバッグ" and "ショルダーバッグ". Determine how existing decomposition handles dictionary sub-tokens and define completion as decomposing applicable compounds into forms such as "トート バッグ" and "ショルダー バッグ", or providing the requested forced-decomposition capability.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.