CharFilter offsets correction is wonky [LUCENE-6595]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Spinoff from this original Elasticsearch issue: https://github.com/elastic/elasticsearch/issues/11726
If I make a MappingCharFilter with these mappings:
```
( ->
) ->
```
i.e., just erase left and right paren, then tokenizing the string
"(F31)" with e.g. WhitespaceTokenizer, produces a single token F31,
with start offset 1 (good).
But for its end offset, I would expect/want 4, but it produces 5
today.
This can be easily explained given how the mapping works: each time a
mapping rule matches, we update the cumulative offset difference,
conceptually as an array like this (it's encoded more compactly):
```
Output offset: 0 1 2 3
Input offset: 1 2 3 5
```
When the tokenizer produces F31, it assigns it startOffset=0 and
endOffset=3 based on the characters it sees (F, 3, 1). It then asks
the CharFilter to correct those offsets, mapping them backwards
through the above arrays, which creates startOffset=1 (good) and
endOffset=5 (bad).
At first, to fix this, I thought this is an "off-by-1" and when
correcting the endOffset we really should return
1+correct(outputEndOffset-1), which would return the correct value (4)
here.
But that's too naive, e.g. here's another example:
```
cccc -> cc
```
If I then tokenize cccc, today we produce the correct offsets (0, 4)
but if we do this "off-by-1" fix for endOffset, we would get the wrong
endOffset (2).
I'm not sure what to do here...
---
Migrated from [LUCENE-6595](https://issues.apache.org/jira/browse/LUCENE-6595) by Michael McCandless (@mikemccand), 2 votes, updated Jul 14 2015
Attachments: [LUCENE-6595.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6595/LUCENE-6595.patch) (versions: 4), [Lucene-6595.pptx](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-6595/Lucene-6595.pptx)
Linked issues:
- #6796
Contributor guide
Research direction
Start by reproducing the offset behavior with MappingCharFilter and WhitespaceTokenizer for the parentheses and cccc mappings. Read the linked issue #6796 and the LUCENE-6595 patch and presentation to understand the proposed direction; done means end offsets are correct for both examples without regressing existing offset correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100