apache / apache/lucene

BaseFragmentsBuilder doesn't calculate the endOffset properly in FastVectorHighlighter(FVH)

Open
#12,221 2 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

This is an issue found in BaseFragmentsBuilder. It makes FVH could potentially append the first word from the next field under a multiValue field to the fragment.

To reproduce it:
1. We need to use FVH to highlight a multiValue field
2. We need to use ScoreFragListBuilder allow the fragment building order is not the indexing order
3. We need to have a document whose best matched fragment is not from the first indexed field
4. We need the fragment size large enough, so the fragment's end offset could be larger than the field end offset
5. We need to fetch not only the top fragments (i.e. maxNumFragments > 1 )
6. FVH fetches the fragments from the fields which are indexed before the best matched field

To make it easier to understand, I created a branch with a [test case](https://github.com/apache/lucene/commit/70aaf684196605c0eef315766383cd8f29d9848f#diff-6e5c26b879d097826a8739ee7ca1efaa4b908d12a5a5d6a8c9fc0c5f146573e9R68).

Note that
The last word "Second" comes from a different field.

### Version and environment details

in master

Contributor guide

Open the contributing guide

Research direction

Start by reading BaseFragmentsBuilder and the linked test case for FastVectorHighlighter with ScoreFragListBuilder and multivalued fields. Run the reproduction with more than one fragment and verify that a fragment does not include text from the next indexed field when its end offset exceeds the field end offset.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.