When the term of more than 16, highlight the query does not return [LUCENE-9609]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
I noticed that when there are too many terms, the highlighted query is restricted
I know that in TermInSetQuery, when there are fewer terms, BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD = 16 will be used to improve query efficiency
```java
static final int BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD = 16;
public Query rewrite(IndexReader reader) throws IOException {
final int threshold = Math.min(BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD, BooleanQuery.getMaxClauseCount());
if (termData.size() <= threshold) {
BooleanQuery.Builder bq = new BooleanQuery.Builder();
TermIterator iterator = termData.iterator();
for (BytesRef term = iterator.next(); term != null; term = iterator.next()) {
bq.add(new TermQuery(new Term(iterator.field(), BytesRef.deepCopyOf(term))), Occur.SHOULD);
}
return new ConstantScoreQuery(bq.build());
}
return super.rewrite(reader);
}
```
When the term of the query statement exceeds 16, the createWeight method in TermInSetQuery will be used
```java
public Weight createWeight(IndexSearcher searcher, boolean needsScores, float boost) throws IOException {
return new ConstantScoreWeight(this, boost) {
`@Override`
public void extractTerms(Set terms) {
// no-op
// This query is for abuse cases when the number of terms is too high to
// run efficiently as a BooleanQuery. So likewise we hide its terms in
// order to protect highlighters
}
......
}
```
I want to ask, why do you say "we hide its terms in order to protect highlighters"
How to implement such " protect highlighters"?
---
Migrated from [LUCENE-9609](https://issues.apache.org/jira/browse/LUCENE-9609) by WangFeiCheng
Contributor guide
Research direction
Start with TermInSetQuery.rewrite and createWeight, then trace how highlighters consume extracted terms. The issue does not name a target file, test, or expected highlighting behavior, so first clarify the desired result for queries with more than 16 terms; done should include agreed behavior and regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100