apache / apache/lucene

Kuromoji tokenizer discards tokens if they start with a punctuation character [LUCENE-9390]

Open
#10,430 10 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This issue was first raised in Elasticsearch [here](https://github.com/elastic/elasticsearch/issues/57614)

The unidic dictionary that is used by the Kuromoji tokenizer contains entries that mix punctuations and other characters. For instance the following entry:

_(株),1285,1285,3690,名詞,一般,**,**,**,**,(株),カブシキガイシャ,カブシキガイシャ_

can be found in the Noun.csv file.

Today, tokens that start with punctuations are automatically removed by default (discardPunctuation  is true). I think the code was written this way because we expect punctuations to be separated from normal tokens but there are exceptions in the original dictionary. Maybe we should check the entire token when discarding punctuations ?

---
Migrated from [LUCENE-9390](https://issues.apache.org/jira/browse/LUCENE-9390) by Jim Ferenczi (@jimczi), updated Jun 19 2020
Pull requests: https://github.com/apache/lucene-solr/pull/1577

Contributor guide

Open the contributing guide

Research direction

Start with the mixed punctuation entry shown from the unidic dictionary's Noun.csv and reproduce how the Kuromoji tokenizer handles it when discardPunctuation is true. Trace the tokenizer's punctuation-discarding path and verify that tokens containing non-punctuation characters are retained while punctuation-only tokens remain discarded.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.