apache / apache/lucene

Highlighter's TokenSources should use TermVector positions if indexed [LUCENE-3091]

Open
#4,164 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor module:highlighter type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Spinoff from #4160

TokenSources is able to use positions indexed into TermVectors, if you pass "false" for the tokenPositionsGuaranteedContiguous argument to getTokenStream.

But if you pass true, it doesn't use the positions. Maybe, we are not allowed (supposed) to pass true if the posIncr could be 0? But, if that is OK/allowed, then we should fix the code to handle the posIncr=0 case correctly.

#4160 made one simple fix, to guess at the posInc (1 or 0) based on the offsets of adjacent tokens.

---
Migrated from [LUCENE-3091](https://issues.apache.org/jira/browse/LUCENE-3091) by Michael McCandless (@mikemccand)

Contributor guide

Open the contributing guide

Research direction

Start with TokenSources.getTokenStream and read the related change in issue #4160. Trace how the tokenPositionsGuaranteedContiguous argument and posIncr=0 are handled, then determine the expected behavior for indexed TermVector positions when the argument is true. Done means the highlighter uses those positions correctly and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.