Highlighter's TokenSources should use TermVector positions if indexed [LUCENE-3091]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Spinoff from #4160
TokenSources is able to use positions indexed into TermVectors, if you pass "false" for the tokenPositionsGuaranteedContiguous argument to getTokenStream.
But if you pass true, it doesn't use the positions. Maybe, we are not allowed (supposed) to pass true if the posIncr could be 0? But, if that is OK/allowed, then we should fix the code to handle the posIncr=0 case correctly.
#4160 made one simple fix, to guess at the posInc (1 or 0) based on the offsets of adjacent tokens.
---
Migrated from [LUCENE-3091](https://issues.apache.org/jira/browse/LUCENE-3091) by Michael McCandless (@mikemccand)
Contributor guide
Research direction
Start with TokenSources.getTokenStream and read the related change in issue #4160. Trace how the tokenPositionsGuaranteedContiguous argument and posIncr=0 are handled, then determine the expected behavior for indexed TermVector positions when the argument is true. Done means the highlighter uses those positions correctly and the behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100