apache / apache/lucene

Highlighter.getBestFragments() merges zero-scored fragments with scored fragments, polluting highlight results

Open
#15,333 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

## Description

The `Highlighter.getBestFragments()` method merges contiguous fragments regardless of score, causing zero-scored (non-matching) fragments to be merged with scored fragments and returned as highlights. This results in large blocks of irrelevant text appearing in highlight results simply because they're adjacent to actual matches.

## Environment
- Lucene version: 10.2.2 (also present in 10.3.1)
- Component: lucene-highlighter

## Steps to Reproduce

1. Create a document with a single term match ("credit") surrounded by substantial text
2. Configure `SimpleSpanFragmenter` with `fragmentSize=100`
3. Call `highlighter.getBestFragments()` with `maxFragments=3`
4. Observe that 14 fragments are created, but only 1 has score > 0
5. The FragmentQueue selects the top 3 fragments (fragment 0 with score 1.0, fragments 1 and 2 with score 0.0)
6. `mergeContiguousFragments()` merges all three into a single ~300 char result

## Actual Behaviour

With `maxFragments=3`, returns a single merged fragment of ~300 characters:

```
@meta name "Process Payment" @meta description "Process a payment for an order using credit card" @meta tags ["payments", "create", "checkout"] @meta collection "Payment Processing API" /* * Payment processing endpoint with PCI compliance * NOTE: All card data must be tokenised before
```

The result includes ~250 characters of zero-scored content merged with the ~50 characters containing the actual match.

## Expected Behaviour

Highlights should only include fragments containing actual matches (score > 0). Zero-scored fragments should either:
1. Not be selected by the FragmentQueue, or
2. Not be merged with scored fragments, or
3. Be filtered out before being returned

Expected result:

```
@meta description "Process a payment for an order using credit card"
```

## Root Cause Analysis

In `getBestTextFragments()`:

1. The fragmenter creates 14 fragments across the document
2. Only 1 fragment contains the search term and has score 1.0
3. The remaining 13 fragments have score 0.0
4. `FragmentQueue(maxNumFragments)` keeps the top N fragments by score
5. Since 13 fragments have identical zero scores, the queue arbitrarily selects the first N-1 zero-scored fragments encountered (fragments 1, 2, etc.)
6. `mergeContiguousFragments()` merges any adjacent fragments regardless of score
7. The merged fragment inherits the highest score (1.0), so it passes the `score > 0` filter

## Problematic Code

In `Highlighter.getBestTextFragments()`, the merge happens unconditionally:

```java
if (mergeContiguousFragments) {
mergeContiguousFragments(frag);
}
```

## Impact

- `maxFragments` parameter behaves counterintuitively - changing it from 2 to 3 changes the result size from ~200 to ~300 chars
- Users get large blocks of irrelevant text in their highlights
- No way to control this behaviour through configuration

## Workaround

Call `getBestTextFragments()` directly with `mergeContiguousFragments=false` and manually filter:

```java
TextFragment[] fragments = highlighter.getBestTextFragments(
tokenStream, text, false, maxFragments);

List results = new ArrayList<>();
for (TextFragment frag : fragments) {
if (frag != null && frag.getScore() > 0) {
results.add(frag.toString());
}
}
```

## Suggested Fix

**Option 1:** Only merge fragments where both fragments have score > threshold (e.g., 0.1)

**Option 2:** Add a configuration parameter to control merge behaviour:

```java
highlighter.setMergeScoreThreshold(0.1);
```

**Option 3:** Filter zero-scored fragments from the FragmentQueue before merging

## Questions

Is the current behaviour intentional? If so, would you consider adding configuration to control which fragments are eligible for merging based on their scores?

### Version and environment details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in Highlighter.getBestTextFragments(), tracing the unconditional mergeContiguousFragments(frag) call and the FragmentQueue selection described in the issue. Reproduce the case with SimpleSpanFragmenter using a fragment size of 100 and maxFragments=3; done means zero-scored fragments no longer enlarge returned highlights around the matching text.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.