apache / apache/lucene

EdgeNGramTokenFilter breaks highlighting by keeping end offsets of original tokens [LUCENE-7758]

Open
#8,809 7 comments 0 reactions 0 assignees View on GitHub
affects-version:6.4.1 legacy-jira-label:EdgeNGramTokenFilter legacy-jira-label:highlighting legacy-jira-priority:Major module:analysis type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When EdgeNGramTokenFilter produces new tokens, they inherit end positions from parent tokens. This behaviour is irrational and breaks highlighting: highlighted not matched pattern, but whole source tokens.

Seems like similar problem was fixed in #4716, but end offsets was broken again after #4980.

Some discussion was found in [SOLR-7926](https://issues.apache.org/jira/browse/SOLR-7926):
> I agree this (highlighting of hits from tokens produced by
> EdgeNGramFilter) got worse with #4980, but it's not clear how to
> fix it\.
> The stacking seems more correct: all these grams are logically
> interchangeable with the original token, and were derived from it, so
> e\.g\. a phrase query involving them with adjacent tokens would work
> correctly\.
> We could perhaps remove the token graph requirement that tokens
> leaving from the same node have the same startOffset, and arriving to
> the same node have the same endOffset\. Lucene would still be able to
> index such a graph, as long as all tokens leaving a given node are
> sorted according to their startOffset\. But I'm not sure if there
> would be other problems\.\.\.
> Or we could maybe improve the token graph, at least for the non-edge
> NGramTokenFilter, so that the grams are linked up correctly, so that any
> path through the graph reconstructs the original characters\.
> But realistically it's not possible to innovate much with token graphs
> in Lucene today because of apparently severe back compat requirements:
> e\.g\. #7722, which fixes the token graph bugs in the existing
> SynonymFilter so that proximity queries work correctly when using
> search-time synonyums, is blocked because of the back compat concerns
> from LUCENE-6721\.
> I'm not sure what the path forward is\.\.\.

---
Migrated from [LUCENE-7758](https://issues.apache.org/jira/browse/LUCENE-7758) by Mikhail Bystryantsev, updated Mar 30 2017
Environment:
```
elasticsearch-5.3
```

Contributor guide

Open the contributing guide

Research direction

Start by inspecting EdgeNGramTokenFilter and the changes referenced in #4716 and #4980. Review the related discussion in SOLR-7926 to understand the token-graph and backward-compatibility constraints. Done means establishing a safe offset behavior that fixes highlighting without breaking phrase queries or other token-graph use cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.