apache / apache/doris

[Enhancement] Support Japanese (Kuromoji) morphological analyzer for inverted index

Open
#64,646 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

Doris inverted index currently has no Japanese-aware tokenizer. Japanese text has no whitespace word boundaries, so the existing english/unicode/standard parsers either index whole runs of text or split on characters, both of which give poor MATCH / MATCH_PHRASE recall and precision for Japanese content.

OpenSearch and Elasticsearch solve this with the Lucene kuromoji morphological analyzer. Doris already ships a comparable CJK analyzer for Chinese aka the IK analyzer `(be/src/storage/index/inverted/analyzer/ik/)`. but there is no equivalent for Japanese.

This enhancement proposes a built-in kuromoji parser, selectable per inverted-indexed column via DDL, that segments Japanese text into morphemes at index and query time:

```
INDEX content_idx (`content`) USING INVERTED
PROPERTIES("parser" = "kuromoji", "parser_mode" = "search")
```

Once indexed, MATCH, MATCH_PHRASE, and TOKENIZE() operate over the segmented Japanese terms.

### Motivation
- Enables accurate full-text search over Japanese columns, on par with OpenSearch/Lucene kuromoji.
- Fills the obvious gap next to the existing Chinese (IK) analyzer.
- Implemented natively in C++ with no JVM on the indexing hot path, and Apache-license-clean (engine is Apache-2.0; the IPADIC dictionary is NAIST-2003, the same permissive lexicon Apache Lucene already
bundles).

### Solution

Add a native C++ port of the Lucene kuromoji analyzer, following the proven IK pattern (native C++ analyzer + tokenizer, dictionary as runtime data files).

- An offline converter that compiles raw IPADIC into a C++-native runtime format, rather than reimplementing Lucene's FST byte format.
- `parser_mode` support: search (default, with SEARCH-mode decompounding), normal, and extended.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing native CJK analyzer under be/src/storage/index/inverted/analyzer/ik/ and trace how its parser is selected from inverted-index DDL. Then investigate the index and query paths used by MATCH, MATCH_PHRASE, and TOKENIZE(). Done means a native Kuromoji parser with runtime dictionary data, the three parser modes, and Japanese terms working consistently at index and query time.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
database, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.