Highlighter.getBestFragments() merges zero-scored fragments with scored fragments, polluting highlight results
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
## Description
The `Highlighter.getBestFragments()` method merges contiguous fragments regardless of score, causing zero-scored (non-matching) fragments to be merged with scored fragments and returned as highlights. This results in large blocks of irrelevant text appearing in highlight results simply because they're adjacent to actual matches.
## Environment
- Lucene version: 10.2.2 (also present in 10.3.1)
- Component: lucene-highlighter
## Steps to Reproduce
1. Create a document with a single term match ("credit") surrounded by substantial text
2. Configure `SimpleSpanFragmenter` with `fragmentSize=100`
3. Call `highlighter.getBestFragments()` with `maxFragments=3`
4. Observe that 14 fragments are created, but only 1 has score > 0
5. The FragmentQueue selects the top 3 fragments (fragment 0 with score 1.0, fragments 1 and 2 with score 0.0)
6. `mergeContiguousFragments()` merges all three into a single ~300 char result
## Actual Behaviour
With `maxFragments=3`, returns a single merged fragment of ~300 characters:
```
@meta name "Process Payment" @meta description "Process a payment for an order using credit card" @meta tags ["payments", "create", "checkout"] @meta collection "Payment Processing API" /* * Payment processing endpoint with PCI compliance * NOTE: All card data must be tokenised before
```
The result includes ~250 characters of zero-scored content merged with the ~50 characters containing the actual match.
## Expected Behaviour
Highlights should only include fragments containing actual matches (score > 0). Zero-scored fragments should either:
1. Not be selected by the FragmentQueue, or
2. Not be merged with scored fragments, or
3. Be filtered out before being returned
Expected result:
```
@meta description "Process a payment for an order using credit card"
```
## Root Cause Analysis
In `getBestTextFragments()`:
1. The fragmenter creates 14 fragments across the document
2. Only 1 fragment contains the search term and has score 1.0
3. The remaining 13 fragments have score 0.0
4. `FragmentQueue(maxNumFragments)` keeps the top N fragments by score
5. Since 13 fragments have identical zero scores, the queue arbitrarily selects the first N-1 zero-scored fragments encountered (fragments 1, 2, etc.)
6. `mergeContiguousFragments()` merges any adjacent fragments regardless of score
7. The merged fragment inherits the highest score (1.0), so it passes the `score > 0` filter
## Problematic Code
In `Highlighter.getBestTextFragments()`, the merge happens unconditionally:
```java
if (mergeContiguousFragments) {
mergeContiguousFragments(frag);
}
```
## Impact
- `maxFragments` parameter behaves counterintuitively - changing it from 2 to 3 changes the result size from ~200 to ~300 chars
- Users get large blocks of irrelevant text in their highlights
- No way to control this behaviour through configuration
## Workaround
Call `getBestTextFragments()` directly with `mergeContiguousFragments=false` and manually filter:
```java
TextFragment[] fragments = highlighter.getBestTextFragments(
tokenStream, text, false, maxFragments);
List results = new ArrayList<>();
for (TextFragment frag : fragments) {
if (frag != null && frag.getScore() > 0) {
results.add(frag.toString());
}
}
```
## Suggested Fix
**Option 1:** Only merge fragments where both fragments have score > threshold (e.g., 0.1)
**Option 2:** Add a configuration parameter to control merge behaviour:
```java
highlighter.setMergeScoreThreshold(0.1);
```
**Option 3:** Filter zero-scored fragments from the FragmentQueue before merging
## Questions
Is the current behaviour intentional? If so, would you consider adding configuration to control which fragments are eligible for merging based on their scores?
### Version and environment details
_No response_
Contributor guide
Research direction
Start in Highlighter.getBestTextFragments(), tracing the unconditional mergeContiguousFragments(frag) call and the FragmentQueue selection described in the issue. Reproduce the case with SimpleSpanFragmenter using a fragment size of 100 and maxFragments=3; done means zero-scored fragments no longer enlarge returned highlights around the matching text.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100