apache / apache/lucene

QueryScorer.init returns empty TokenStream if setMaxDocCharsToAnalyze is not previously called [LUCENE-7038]

Open
#8,095 0 comments 0 reactions 0 assignees View on GitHub
affects-version:5.4 legacy-jira-priority:Minor module:highlighter type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

This is a regression since Lucene 4.10 regarding The QueryScorer class in the Highlighter module.

In 4.10, the `QueryScorer.init` method returns a working tokenStream even if the maxCharsToAnalyze variable is set to zero. In both versions, zero is the default value and in 4.10 it indicated that the entire stream should be returned, not an empty stream.

The problem is with the `WeightedSpanTermExtractor` always wrapping the tokenStream in a `OffsetLimitTokenFilter` filter, even when the passed down maxDocCharsToAnalyze variable is zero.

---
Migrated from [LUCENE-7038](https://issues.apache.org/jira/browse/LUCENE-7038) by Jeff Stein, updated Feb 20 2016
Linked issues:
- #7435

Contributor guide

Open the contributing guide

Research direction

Start with QueryScorer.init in the Highlighter module and trace how maxDocCharsToAnalyze reaches WeightedSpanTermExtractor and its OffsetLimitTokenFilter wrapping. Verify the zero-default behavior against the Lucene 4.10 behavior described in the issue; done means a zero limit returns the complete token stream while a nonzero limit remains bounded.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.