apache / apache/lucene

Add `keepDecimalPoint` configuration option to KoreanTokenizer Nori

Open
#15,894 0 comments 0 reactions 0 assignees View on GitHub
type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

## Description
Currently, the `KoreanTokenizer` in the Nori module segments numeric values containing decimal points into multiple separate tokens.

For example, the phrase **"10.1인치 모니터"** is currently tokenized as follows (when `discardPunctuation = false`):
`["10", ".", "1", "인치", "모니터"]`

**Why this is important (for example: Media/News Use Cases)**
In news datasets, numeric precision is vital for accurate information retrieval. Key values such as **economic growth rates ("3.5%"), poll results ("42.1%"), or exchange rates ("1532.26원")** are currently fragmented.

This leads to poor search relevancy for data-driven reporting, as it becomes difficult to search for specific numeric measurements or versions (e.g., searching for "10.1" might match "10" and "1" separately). Treating these as atomic units is essential for maintaining the semantic integrity of the data.

## Proposed Enhancement
I propose adding a new configuration option, `keepDecimalPoint` (boolean), to the `KoreanTokenizer`.

- Behavior: When set to `true`, the tokenizer treats a sequence of `[Digit][Dot][Digit]` as a single token during the lattice construction phase.
- Default value: `false` (to maintain backward compatibility).

## Example of desired output (`keepDecimalPoint=true`)
- Input: `"10.1인치 모니터"`
- Output: `["10.1", "인치", "모니터"]`

Contributor guide

Open the contributing guide

Research direction

Start at the KoreanTokenizer in the Nori module and trace its lattice construction for numeric tokens. Inspect the existing tokenizer tests, if present, before defining coverage for the new option. Done means keepDecimalPoint defaults to false while true preserves digit-dot-digit sequences such as 10.1 as one token.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.