Build kuromoji system dictionary as a separated jar and load it from JapaneseTokenizer at runtime [LUCENE-8869]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This is a sub-task for #9860.
In this issue, I will try to make small but self-contained changes to kuromoji system dictionary.
- Make it possible to build a jar that contains (maybe) only dictionary data resource generated by the `build-dict` task.
- Maybe a new ant target will be added.
- Make it possible to load external dictionary when initializing JapaneseTokenizer.
- Some work are already done on #9906
- Decouple current system dictionary data (mecab ipadic) from kuromoji itself and use it as default (Possibly it can be done with another issue).
Also, some refactoring of the directory/source tree structure may be needed.
---
Migrated from [LUCENE-8869](https://issues.apache.org/jira/browse/LUCENE-8869) by Tomoko Uchida (@mocobeta), 1 vote, updated Jun 23 2019
Linked issues:
- #9860
Contributor guide
Research direction
Start by reading the parent issue #9860 and related work in #9906, then inspect the kuromoji `build-dict` task and `JapaneseTokenizer` initialization. The scope includes a separate dictionary-data jar, runtime loading of an external dictionary, and possible source-tree refactoring; confirm which parts remain before defining completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100