apache / apache/lucene

UnifiedHighlighter not highlighting all terms relevant in SpanNearQuery [LUCENE-7682]

Open
#8,733 5 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major module:highlighter type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Original text: "Something for protecting wildlife feed in a feed thing."
Query is:
SpanNearQuery with Slop 9 - in order -
1. SpanTermQuery(wildlife)
2. SpanTermQuery(feed)

This should highlight both instances of "feed" since they are both within slop of 9 of "wildlife". However, only the first instance is highlighted. This occurs with unordered SpanNearQuery as well. Test below replicates. Affects both the current 6.x line and master.

Test that fits within TestUnifiedHighlighterMTQ:

```Java
public void testOrderedSpanNearQueryWithDupeTerms() throws Exception {
RandomIndexWriter iw = new RandomIndexWriter(random(), dir, indexAnalyzer);
Document doc = new Document();
doc.add(new Field("body", "Something for protecting wildlife feed in a feed thing.", fieldType));
doc.add(newTextField("id", "id", Field.Store.YES));

iw.addDocument(doc);
IndexReader ir = iw.getReader();
iw.close();

IndexSearcher searcher = newSearcher(ir);
UnifiedHighlighter highlighter = new UnifiedHighlighter(searcher, indexAnalyzer);
int docID = searcher.search(new TermQuery(new Term("id", "id")), 1).scoreDocs[0].doc;

SpanTermQuery termOne = new SpanTermQuery(new Term("body", "wildlife"));
SpanTermQuery termTwo = new SpanTermQuery(new Term("body", "feed"));
SpanNearQuery topQuery = new SpanNearQuery.Builder("body", true)
.setSlop(9)
.addClause(termOne)
.addClause(termTwo)
.build();

int[] docIds = new int[] {docID};

String snippets[] = highlighter.highlightFields(new String[] {"body"}, topQuery, docIds, new int[] {2}).get("body");
assertEquals(1, snippets.length);
assertEquals("Something for protecting wildlife feed in a feed thing.", snippets[0]);
ir.close();
}
```

---
Migrated from [LUCENE-7682](https://issues.apache.org/jira/browse/LUCENE-7682) by Michael Braun (@michaelbraun), updated Aug 16 2017
Linked issues:
- #7595

Contributor guide

Open the contributing guide

Research direction

Start by locating and running the TestUnifiedHighlighterMTQ test mentioned in the issue, especially the provided ordered SpanNearQuery case. Trace the UnifiedHighlighter behavior for duplicate terms within the span query. Done means the test passes and both occurrences of “feed” are highlighted in the expected snippet.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.