apache / apache/lucene

UnifiedHighlighter incorrectly returns field 'X' was indexed without offsets

Open
#13,103 2 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

UnifiedHighlighter based on matches incorrectly returns field 'X' was indexed without offsets, cannot highlight

Test to reproduce:
```java
static final FieldType textType = new FieldType(TextField.TYPE_STORED);
static {
textType.setStoreTermVectors(true);
textType.setStoreTermVectorPositions(true);
textType.setStoreTermVectorOffsets(true);
textType.freeze();
}

public void testHighlgiht() {
String indexPath = "../lucene-test-indices/index1";
Path path = Paths.get(indexPath);
try {
Directory directory = NIOFSDirectory.open(path);
Analyzer analyzer = new ClassicAnalyzer();
IndexWriterConfig config = new IndexWriterConfig(analyzer);

try (IndexWriter writer = new IndexWriter(directory, config)) {
addDoc(writer, "The quick brown fox jumps over the lazy dog");
}

try (IndexReader reader = DirectoryReader.open(directory)) {
IndexSearcher searcher = new IndexSearcher(reader);
Query query = new IntervalQuery("content",
Intervals.analyzedText("quick brown fox jumps over the lazy dog", analyzer, "content", 0, true));
TopDocs topDocs = searcher.search(query, 10);

UnifiedHighlighter.Builder uhBuilder = new UnifiedHighlighter.Builder(searcher, analyzer)
.withWeightMatches(true);
UnifiedHighlighter highlighter = new UnifiedHighlighter(uhBuilder);

String[] highlights = highlighter.highlight("content", query, topDocs, 1);
System.out.println(Arrays.toString(highlights));
}
} catch (IOException e) {
e.printStackTrace();
}
}

private static void addDoc(IndexWriter writer, String content) throws IOException {
Document doc = new Document();
doc.add(new Field("content", content, textType));
writer.addDocument(doc);
}
```

produces an error:
```
java.lang.IllegalArgumentException: field 'content' was indexed without offsets, cannot highlight

at org.apache.lucene.search.uhighlight.FieldHighlighter.highlightOffsetsEnums(FieldHighlighter.java:157)
at org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFieldsAsObjects(UnifiedHighlighter.java:944)
at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFields(UnifiedHighlighter.java:814)
at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlightFields(UnifiedHighlighter.java:792)
at org.apache.lucene.search.uhighlight.UnifiedHighlighter.highlight(UnifiedHighlighter.java:725)
```

A workaround to disable highlighting based on matches:

```java
UnifiedHighlighter.Builder uhBuilder = new UnifiedHighlighter.Builder(searcher, analyzer)
.withWeightMatches(false);
```

This happens because of `ClassicAnalyzer` that removes stop words, and because of it usage of `ExtendedIntervalsSource` that returns -1 offsets.

### Version and environment details

Lucene v 9.9.1

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied reproduction with Lucene 9.9.1 and inspect UnifiedHighlighter with weight matches enabled. Trace the failure through FieldHighlighter.highlightOffsetsEnums and the ExtendedIntervalsSource behavior when ClassicAnalyzer removes stop words. Done means the query can be highlighted without the incorrect “indexed without offsets” exception while retaining the expected highlighting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.