apache / apache/lucene

UnifiedHighlighter ANALYSIS mode does not accurately highlight SpanNotQuery or MUST_NOT [LUCENE-9426]

Open
#10,466 2 comments 0 reactions 0 assignees View on GitHub
affects-version:8.5.1 legacy-jira-label:easyfix legacy-jira-priority:Major module:highlighter type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

If UnifiedHighlighter uses MemoryIndexOffsetStrategy, it does not treat SpanNotQuery correctly.
Since UnifiedHighlighter uses actual search in order to determine which locations to highlight, it should be consistent with search and only highlight locations in a document that really match the query. However, it does not for SpanNotQuery.

For the query spanNot(spanNear([content:100, content:dollars], 1, true), content:thousand, 0, 0)
it produces
A <b>100</b> fucking <b>dollars</b> wasn't enough to fix it. ... We need <b>100</b> thousand <b>dollars</b> to buy the house

---
Migrated from [LUCENE-9426](https://issues.apache.org/jira/browse/LUCENE-9426) by Christoph Goller
Environment:
```
I tested with 8.5.1, but other versions are probably also affected.
```

Attachments: [TestUnifiedHighlighter.java](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9426/TestUnifiedHighlighter.java)

Contributor guide

Open the contributing guide

Research direction

Start with the attached TestUnifiedHighlighter.java and reproduce the SpanNotQuery example using UnifiedHighlighter in ANALYSIS mode with MemoryIndexOffsetStrategy. Trace how the query determines highlighted locations, then verify the result against actual search matching. Done means SpanNotQuery and MUST_NOT highlights include only locations that truly match the query.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.