Kuromoji tokenizer discards tokens if they start with a punctuation character [LUCENE-9390]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This issue was first raised in Elasticsearch [here](https://github.com/elastic/elasticsearch/issues/57614)
The unidic dictionary that is used by the Kuromoji tokenizer contains entries that mix punctuations and other characters. For instance the following entry:
_(株),1285,1285,3690,名詞,一般,**,**,**,**,(株),カブシキガイシャ,カブシキガイシャ_
can be found in the Noun.csv file.
Today, tokens that start with punctuations are automatically removed by default (discardPunctuation is true). I think the code was written this way because we expect punctuations to be separated from normal tokens but there are exceptions in the original dictionary. Maybe we should check the entire token when discarding punctuations ?
---
Migrated from [LUCENE-9390](https://issues.apache.org/jira/browse/LUCENE-9390) by Jim Ferenczi (@jimczi), updated Jun 19 2020
Pull requests: https://github.com/apache/lucene-solr/pull/1577
Contributor guide
Research direction
Start with the mixed punctuation entry shown from the unidic dictionary's Noun.csv and reproduce how the Kuromoji tokenizer handles it when discardPunctuation is true. Trace the tokenizer's punctuation-discarding path and verify that tokens containing non-punctuation characters are retained while punctuation-only tokens remain discarded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100