apache / apache/lucene

[Nori] Add metadata support for Korean analyzer tokens

Open
#14,940 0 comments 0 reactions 0 assignees View on GitHub
type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

## Overview
This issue proposes adding metadata support to the Nori Korean analyzer, allowing users to attach additional information to dictionary words that can be accessed during text analysis.

## Background and Motivation
Currently, the Nori analyzer allows users to register words in a custom dictionary, but there's no way to associate additional information with these words. By supporting metadata, we can enable:
- Attaching semantic category information to words (e.g., "Java" -> "programming language")
- Preserving information for compound words
- Custom tagging and classification
- Domain-specific annotations

## Proposed Implementation
1. Add metadata support to Token class
2. Create `MetadataAttribute` and its implementation
3. Extend user dictionary format with a metadata separator (`>>`)
4. Preserve metadata during compound word decomposition

## Usage Example
User dictionary:
```
자바 >> computer language
java >> computer language
엘라스틱서치 엘라스틱 서치 >> search engine
```

and this should be

input : 자바
```
/* Output:
Term: 자바
Metadata: computer language
POS: NNG
---
```

input : 엘라스틱서치
```
/* Output:
Term: 엘라스틱서치
Metadata: search engine
Position Increment: 1
Position Length: 2
---
Term: 엘라스틱
Metadata: search engine
Position Increment: 0
Position Length: 1
---
Term: 서치
Metadata: search engine
Position Increment: 1
Position Length: 1
---
```

## Benefits
1. Enhanced information modeling: Attach additional information to words to improve search quality
2. Domain-specific analysis: Define metadata relevant to specific domains
3. Custom dictionary extension: Add capabilities while maintaining backward compatibility

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Nori analyzer's Token class, the proposed MetadataAttribute implementation, and the user dictionary parsing and compound-word decomposition paths. Define how the `>>` separator remains backward compatible, then verify metadata for simple and compound entries using the usage examples; no specific files or tests are named.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.