SegmentingTokenizerBase splits terms that occupy 1024th positions in text [LUCENE-9112]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
The OpenNLP tokenizer show weird behaviour when text contains spurious punctuation such as having triple dots trailing a sentence...
1. the first dot becomes part of the token, having 'sentence.' becomes the token
1. much further down the text, a seemingly unrelated token is then suddenly split up, in my example (see attached unit test) the name 'Baron' is split into 'Baro' and 'n', this is the real problem
The problems never seem to occur when using small texts in unit tests but it certainly does in real world examples. Depending on how many 'spurious' dots, a completely different term can become split, or the same term in just a different location.
I am not too sure if this is actually a problem in the Lucene code, but it is a problem and i have a Lucene unit test proving the problem.
---
Migrated from [LUCENE-9112](https://issues.apache.org/jira/browse/LUCENE-9112) by Markus Jelsma, updated Dec 08 2021
Attachments: [en-sent.bin](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9112/en-sent.bin), [en-token.bin](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9112/en-token.bin), [LUCENE-9112.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9112/LUCENE-9112.patch) (versions: 2), [LUCENE-9112-unittest.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9112/LUCENE-9112-unittest.patch) (versions: 2)
Contributor guide
Research direction
Start with the attached Lucene unit-test patch and reproduce the OpenNLP tokenizer behavior around the 1024th text position, using the en-sent.bin and en-token.bin attachments as needed. The fix is complete when the reported punctuation case no longer splits a later term such as “Baron” and the regression test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100