apache / apache/lucene

JapaneseTokenizer: Validate segmentation of User Dictionary entries on creation [LUCENE-7181]

Open
#8,236 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

From the [conversation on the dev list](http://mail-archives.apache.org/mod_mbox/lucene-dev/201604.mbox/%3CCAMJgJxR8gLnXi7WXkN3KFfxHu=PoSeVxXARbbg+cHCE1TZhhpQ@mail.gmail.com%3E)

The user dictionary in the `JapaneseTokenizer` allows users to customize how a stream is broken into tokens using a specific set of rules provided like:
AABBBCC -> AA BBB CC

It does not allow users to change any of the token characters like:

(1) AABBBCC -> DD BBB CC (this will just tokenize to "AA", "BBB", "CC", seems to only care about positions)
It also doesn't let a character be part of more than one token, like:

(2) AABBBCC -> AAB BBB BCC (this will throw an AIOOBE)

..or make the output token bigger than the input text:

(3) AA -> AAA (Also AIOOBE)

Currently there is no validation for those cases, case 1 doesn't fail but provide unexpected tokens. Cases 2 and 3 fail when the input text is analyzed. We should add validation to the `UserDictionary` creation.

---
Migrated from [LUCENE-7181](https://issues.apache.org/jira/browse/LUCENE-7181) by Tomas Eduardo Fernandez Lobbe (@tflobbe), updated Apr 08 2016
Attachments: [LUCENE-7181.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7181/LUCENE-7181.patch)

Contributor guide

Open the contributing guide

Research direction

Start at UserDictionary creation and trace how JapaneseTokenizer consumes the resulting segmentation rules. Review the migrated LUCENE-7181 discussion and attached patch for the expected validation behavior, then verify that the three described invalid mappings are rejected before analysis and that valid entries still segment correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.