QueryScorer.init returns empty TokenStream if setMaxDocCharsToAnalyze is not previously called [LUCENE-7038]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
This is a regression since Lucene 4.10 regarding The QueryScorer class in the Highlighter module.
In 4.10, the `QueryScorer.init` method returns a working tokenStream even if the maxCharsToAnalyze variable is set to zero. In both versions, zero is the default value and in 4.10 it indicated that the entire stream should be returned, not an empty stream.
The problem is with the `WeightedSpanTermExtractor` always wrapping the tokenStream in a `OffsetLimitTokenFilter` filter, even when the passed down maxDocCharsToAnalyze variable is zero.
---
Migrated from [LUCENE-7038](https://issues.apache.org/jira/browse/LUCENE-7038) by Jeff Stein, updated Feb 20 2016
Linked issues:
- #7435
Contributor guide
Research direction
Start with QueryScorer.init in the Highlighter module and trace how maxDocCharsToAnalyze reaches WeightedSpanTermExtractor and its OffsetLimitTokenFilter wrapping. Verify the zero-default behavior against the Lucene 4.10 behavior described in the issue; done means a zero limit returns the complete token stream while a nonzero limit remains bounded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100