EdgeNGramTokenFilter breaks highlighting by keeping end offsets of original tokens [LUCENE-7758]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
When EdgeNGramTokenFilter produces new tokens, they inherit end positions from parent tokens. This behaviour is irrational and breaks highlighting: highlighted not matched pattern, but whole source tokens.
Seems like similar problem was fixed in #4716, but end offsets was broken again after #4980.
Some discussion was found in [SOLR-7926](https://issues.apache.org/jira/browse/SOLR-7926):
> I agree this (highlighting of hits from tokens produced by
> EdgeNGramFilter) got worse with #4980, but it's not clear how to
> fix it\.
> The stacking seems more correct: all these grams are logically
> interchangeable with the original token, and were derived from it, so
> e\.g\. a phrase query involving them with adjacent tokens would work
> correctly\.
> We could perhaps remove the token graph requirement that tokens
> leaving from the same node have the same startOffset, and arriving to
> the same node have the same endOffset\. Lucene would still be able to
> index such a graph, as long as all tokens leaving a given node are
> sorted according to their startOffset\. But I'm not sure if there
> would be other problems\.\.\.
> Or we could maybe improve the token graph, at least for the non-edge
> NGramTokenFilter, so that the grams are linked up correctly, so that any
> path through the graph reconstructs the original characters\.
> But realistically it's not possible to innovate much with token graphs
> in Lucene today because of apparently severe back compat requirements:
> e\.g\. #7722, which fixes the token graph bugs in the existing
> SynonymFilter so that proximity queries work correctly when using
> search-time synonyums, is blocked because of the back compat concerns
> from LUCENE-6721\.
> I'm not sure what the path forward is\.\.\.
---
Migrated from [LUCENE-7758](https://issues.apache.org/jira/browse/LUCENE-7758) by Mikhail Bystryantsev, updated Mar 30 2017
Environment:
```
elasticsearch-5.3
```
Contributor guide
Research direction
Start by inspecting EdgeNGramTokenFilter and the changes referenced in #4716 and #4980. Review the related discussion in SOLR-7926 to understand the token-graph and backward-compatibility constraints. Done means establishing a safe offset behavior that fixes highlighting without breaking phrase queries or other token-graph use cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100