apache / apache/lucene

FastVectorHighlighter ignores position in PhraseQuery [LUCENE-7551]

Open
#8,602 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:highlighter type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The PhraseQuery contains a list of terms and the positions for each term. The FVH ignores the term position and assumes that a phrase query is always dense. As a result phrase query with gaps are not highlighted at all. This is problematic for text fields that use a FilteringTokenFilter. This token filter removes tokens but preserves the position increment of each removal.
Bottom line is that using this token filter breaks the highlighting of phrase query that contains filtered tokens.

---
Migrated from [LUCENE-7551](https://issues.apache.org/jira/browse/LUCENE-7551) by Jim Ferenczi (@jimczi), 1 vote
Linked issues:
- [SOLR-3724](https://issues.apache.org/jira/browse/SOLR-3724)
- #8603

Contributor guide

Open the contributing guide

Research direction

Start by tracing the FastVectorHighlighter handling of PhraseQuery term positions, then reproduce the case with a FilteringTokenFilter that removes tokens while preserving position increments. Done means phrase queries containing filtered-token gaps are highlighted correctly rather than being omitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.