[Nori] Add metadata support for Korean analyzer tokens
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
## Overview
This issue proposes adding metadata support to the Nori Korean analyzer, allowing users to attach additional information to dictionary words that can be accessed during text analysis.
## Background and Motivation
Currently, the Nori analyzer allows users to register words in a custom dictionary, but there's no way to associate additional information with these words. By supporting metadata, we can enable:
- Attaching semantic category information to words (e.g., "Java" -> "programming language")
- Preserving information for compound words
- Custom tagging and classification
- Domain-specific annotations
## Proposed Implementation
1. Add metadata support to Token class
2. Create `MetadataAttribute` and its implementation
3. Extend user dictionary format with a metadata separator (`>>`)
4. Preserve metadata during compound word decomposition
## Usage Example
User dictionary:
```
자바 >> computer language
java >> computer language
엘라스틱서치 엘라스틱 서치 >> search engine
```
and this should be
input : 자바
```
/* Output:
Term: 자바
Metadata: computer language
POS: NNG
---
```
input : 엘라스틱서치
```
/* Output:
Term: 엘라스틱서치
Metadata: search engine
Position Increment: 1
Position Length: 2
---
Term: 엘라스틱
Metadata: search engine
Position Increment: 0
Position Length: 1
---
Term: 서치
Metadata: search engine
Position Increment: 1
Position Length: 1
---
```
## Benefits
1. Enhanced information modeling: Attach additional information to words to improve search quality
2. Domain-specific analysis: Define metadata relevant to specific domains
3. Custom dictionary extension: Add capabilities while maintaining backward compatibility
Contributor guide
Research direction
Start by reviewing the Nori analyzer's Token class, the proposed MetadataAttribute implementation, and the user dictionary parsing and compound-word decomposition paths. Define how the `>>` separator remains backward compatible, then verify metadata for simple and compound entries using the usage examples; no specific files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100