apache / apache/lucene

part of speech tagging for Korean, Japanese [LUCENE-10049]

Open
#11,087 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-label:newbie legacy-jira-priority:Trivial module:analysis type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Korean(nori) and Japanese(kuromoji) analyzers behave the same way by using a dictionary-based and finite-state-based approach to identify words (aka tokens).

When analyzing Korean/Japanese inputs, it needs to perform a lookup in the dictionary on every character in order to build the lattice of all possible segmentations. In order to achieve this efficiently, we encode the full vocabulary in FST (finite state transducer). So we can analyze text using the Viterbi algorithm to find the most likely segmentation (called the Viterbi path) of any input written in Korean or Japanese.

 

```java
org.apache.lucene.analysis.ko.GraphvizFormatter
org.apache.lucene.analysis.ja.GraphvizFormatter
```

 

Those two already have Graphviz to visualize the Viterbi lattice built from input texts. However, according to my experience, part of speech is significant to diagnose why the outputs look like since this works with the dictionary-based approach. 

Adding tokens' part of speech will help users to understand the analyzers. I and some users are using those classes during their Lucene-related projects although it's a very trivial part. Opening a PR after issue review.

---
Migrated from [LUCENE-10049](https://issues.apache.org/jira/browse/LUCENE-10049) by Uihyun Kim (@uihyun)
Attachments: [LUCENE-10049.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-10049/LUCENE-10049.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reading org.apache.lucene.analysis.ko.GraphvizFormatter and org.apache.lucene.analysis.ja.GraphvizFormatter, then trace how each formatter receives the Viterbi lattice from its analyzer. Add part-of-speech information to the visualized tokens, and verify that Korean and Japanese graphs expose the relevant dictionary metadata for diagnosing segmentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.