apache / apache/lucene

Build kuromoji system dictionary as a separated jar and load it from JapaneseTokenizer at runtime [LUCENE-8869]

Open
#9,912 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This is a sub-task for #9860.
In this issue, I will try to make small but self-contained changes to kuromoji system dictionary.
- Make it possible to build a jar that contains (maybe) only dictionary data resource generated by the `build-dict` task.
- Maybe a new ant target will be added.
- Make it possible to load external dictionary when initializing JapaneseTokenizer.
- Some work are already done on #9906
- Decouple current system dictionary data (mecab ipadic) from kuromoji itself and use it as default (Possibly it can be done with another issue).

Also, some refactoring of the directory/source tree structure may be needed.

---
Migrated from [LUCENE-8869](https://issues.apache.org/jira/browse/LUCENE-8869) by Tomoko Uchida (@mocobeta), 1 vote, updated Jun 23 2019
Linked issues:
- #9860

Contributor guide

Open the contributing guide

Research direction

Start by reading the parent issue #9860 and related work in #9906, then inspect the kuromoji `build-dict` task and `JapaneseTokenizer` initialization. The scope includes a separate dictionary-data jar, runtime loading of an external dictionary, and possible source-tree refactoring; confirm which parts remain before defining completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.