NGramTokenFilter may generate offsets that exceed the length of original text [LUCENE-2909]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Whan using NGramTokenFilter combined with CharFilters that lengthen the original text (such as "ß" -> "ss"), the generated offsets exceed the length of the origianal text.
This causes InvalidTokenOffsetsException when you try to highlight the text in Solr.
While it is not possible to know the accurate offset of each character once you tokenize the whole text with tokenizers like KeywordTokenizer, NGramTokenFilter should at least avoid generating invalid offsets.
---
Migrated from [LUCENE-2909](https://issues.apache.org/jira/browse/LUCENE-2909) by Shinya Kasatani, updated May 16 2011
Attachments: [LUCENE-2909_assert.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2909/LUCENE-2909_assert.patch), [TokenFilterOffset.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2909/TokenFilterOffset.patch)
Contributor guide
Assessment
This issue has not been assessed yet.