BaseFragmentsBuilder doesn't calculate the endOffset properly in FastVectorHighlighter(FVH)
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
This is an issue found in BaseFragmentsBuilder. It makes FVH could potentially append the first word from the next field under a multiValue field to the fragment.
To reproduce it:
1. We need to use FVH to highlight a multiValue field
2. We need to use ScoreFragListBuilder allow the fragment building order is not the indexing order
3. We need to have a document whose best matched fragment is not from the first indexed field
4. We need the fragment size large enough, so the fragment's end offset could be larger than the field end offset
5. We need to fetch not only the top fragments (i.e. maxNumFragments > 1 )
6. FVH fetches the fragments from the fields which are indexed before the best matched field
To make it easier to understand, I created a branch with a [test case](https://github.com/apache/lucene/commit/70aaf684196605c0eef315766383cd8f29d9848f#diff-6e5c26b879d097826a8739ee7ca1efaa4b908d12a5a5d6a8c9fc0c5f146573e9R68).
Note that
The last word "Second" comes from a different field.
### Version and environment details
in master
Contributor guide
Research direction
Start by reading BaseFragmentsBuilder and the linked test case for FastVectorHighlighter with ScoreFragListBuilder and multivalued fields. Run the reproduction with more than one fragment and verify that a fragment does not include text from the next indexed field when its end offset exceeds the field end offset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100