JapaneseTokenizer: Validate segmentation of User Dictionary entries on creation [LUCENE-7181]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
From the [conversation on the dev list](http://mail-archives.apache.org/mod_mbox/lucene-dev/201604.mbox/%3CCAMJgJxR8gLnXi7WXkN3KFfxHu=PoSeVxXARbbg+cHCE1TZhhpQ@mail.gmail.com%3E)
The user dictionary in the `JapaneseTokenizer` allows users to customize how a stream is broken into tokens using a specific set of rules provided like:
AABBBCC -> AA BBB CC
It does not allow users to change any of the token characters like:
(1) AABBBCC -> DD BBB CC (this will just tokenize to "AA", "BBB", "CC", seems to only care about positions)
It also doesn't let a character be part of more than one token, like:
(2) AABBBCC -> AAB BBB BCC (this will throw an AIOOBE)
..or make the output token bigger than the input text:
(3) AA -> AAA (Also AIOOBE)
Currently there is no validation for those cases, case 1 doesn't fail but provide unexpected tokens. Cases 2 and 3 fail when the input text is analyzed. We should add validation to the `UserDictionary` creation.
---
Migrated from [LUCENE-7181](https://issues.apache.org/jira/browse/LUCENE-7181) by Tomas Eduardo Fernandez Lobbe (@tflobbe), updated Apr 08 2016
Attachments: [LUCENE-7181.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7181/LUCENE-7181.patch)
Contributor guide
Research direction
Start at UserDictionary creation and trace how JapaneseTokenizer consumes the resulting segmentation rules. Review the migrated LUCENE-7181 discussion and attached patch for the expected validation behavior, then verify that the three described invalid mappings are rejected before analysis and that valid entries still segment correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100