apache / apache/lucene

Combine Nori and Kuromoji DictionaryBuilder [LUCENE-8817]

Open
#9,861 8 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This issue is related to #9860.

Currently Nori and Kuromoji Analyzer use the same dictionary structure. (MeCab)
If we make combine DictionaryBuilder, we can reduce the code size.
But this task may have a dependency on the language.
(like HEADER string in BinaryDictionary and CharacterDefinition, methods in BinaryDictionaryWriter, ...)
On the other hand, there are many overlapped classes.

The purpose of this patch is to provide users of Nori and Kuromoji with the same system dictionary generator.

It may take some time because there is a little workload.
The work will be based on the latest master, and if the #9860 is finished first, I will pull the latest code and proceed.

---
Migrated from [LUCENE-8817](https://issues.apache.org/jira/browse/LUCENE-8817) by Namgyu Kim (@danmuzi), updated Jun 10 2019
Linked issues:
- #9860

Contributor guide

Open the contributing guide

Research direction

Compare the Nori and Kuromoji DictionaryBuilder implementations and their shared MeCab dictionary structure. Start with BinaryDictionary, CharacterDefinition, and BinaryDictionaryWriter, while checking the dependency on #9860 and the latest master. Done means both analyzers use one system dictionary generator without losing their language-specific behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.