apache / apache/lucene

When the term of more than 16, highlight the query does not return [LUCENE-9609]

Open
#10,649 0 comments 0 reactions 0 assignees View on GitHub
affects-version:7.7.3 legacy-jira-priority:Minor module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I noticed that when there are too many terms, the highlighted query is restricted

I know that in TermInSetQuery, when there are fewer terms, BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD = 16 will be used to improve query efficiency

```java
static final int BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD = 16;

public Query rewrite(IndexReader reader) throws IOException {
final int threshold = Math.min(BOOLEAN_REWRITE_TERM_COUNT_THRESHOLD, BooleanQuery.getMaxClauseCount());
if (termData.size() <= threshold) {
BooleanQuery.Builder bq = new BooleanQuery.Builder();
TermIterator iterator = termData.iterator();
for (BytesRef term = iterator.next(); term != null; term = iterator.next()) {
bq.add(new TermQuery(new Term(iterator.field(), BytesRef.deepCopyOf(term))), Occur.SHOULD);
}
return new ConstantScoreQuery(bq.build());
}
return super.rewrite(reader);
}
```

 When the term of the query statement exceeds 16, the createWeight method in TermInSetQuery will be used

```java
public Weight createWeight(IndexSearcher searcher, boolean needsScores, float boost) throws IOException {
return new ConstantScoreWeight(this, boost) {

`@Override`
public void extractTerms(Set terms) {
// no-op
// This query is for abuse cases when the number of terms is too high to
// run efficiently as a BooleanQuery. So likewise we hide its terms in
// order to protect highlighters
}

......
}
```

I want to ask, why do you say "we hide its terms in order to protect highlighters"

How to implement such " protect highlighters"?

---
Migrated from [LUCENE-9609](https://issues.apache.org/jira/browse/LUCENE-9609) by WangFeiCheng

Contributor guide

Open the contributing guide

Research direction

Start with TermInSetQuery.rewrite and createWeight, then trace how highlighters consume extracted terms. The issue does not name a target file, test, or expected highlighting behavior, so first clarify the desired result for queries with more than 16 terms; done should include agreed behavior and regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.