apache / apache/lucene

IndexOrDocValuesQuery is counted twice when computing `maxClauseCount`

Open
#14,756 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I noticed that if I change a query from a PointRangeQuery to an IndexOrDocValuesQuery, then the change might make previous valid boolean queries to fail because IndexOrDocValuesQuery counts twice when computing maxClauseCount.

I think this is wrong as IndexOrDocValues is just providing lucene with the choice of two execution paths and should not count twice for that limit.

This test proves the issue:

```
public void testIndexOrDocValues() throws IOException {
Directory dir = newDirectory();
RandomIndexWriter writer = new RandomIndexWriter(random(), dir);
Document d = new Document();
d.add(new LongField("foo", 0L, LongField.Store.NO));
writer.addDocument(d);
d = new Document();
d.add(new LongField("foo", Long.MAX_VALUE, LongField.Store.NO));
writer.addDocument(d);

IndexReader reader = writer.getReader();
IndexSearcher searcher = newSearcher(reader);
writer.close();
int maxClauseCount = IndexSearcher.getMaxClauseCount();
BooleanQuery.Builder qb = new BooleanQuery.Builder();

for (int i = 0; i < maxClauseCount; i++) {
qb.add(LongPoint.newRangeQuery("foo", 0, i), BooleanClause.Occur.SHOULD);
}
// should not throw an exception, because it is below the limit
searcher.rewrite(qb.build());

qb = new BooleanQuery.Builder();
for (int i = 0; i < maxClauseCount; i++) {
qb.add(LongField.newRangeQuery("foo", 0, i), BooleanClause.Occur.SHOULD);
}
// should not throw an exception, because it is below the limit
searcher.rewrite(qb.build());

reader.close();
dir.close();
}
```

It fails in the second rewrite with the exception:

```
org.apache.lucene.search.IndexSearcher$TooManyNestedClauses: Query contains too many nested clauses; maxClauseCount is set to 1024
at __randomizedtesting.SeedInfo.seed([1DBE2F81F126780A:AB3F95EEC9898FF7]:0)
at org.apache.lucene.search.IndexSearcher$1.visitLeaf(IndexSearcher.java:905)
at org.apache.lucene.document.SortedNumericDocValuesRangeQuery.visit(SortedNumericDocValuesRangeQuery.java:74)
at org.apache.lucene.search.IndexOrDocValuesQuery.visit(IndexOrDocValuesQuery.java:134)
at org.apache.lucene.search.BooleanQuery.visit(BooleanQuery.java:661)
```

Contributor guide

Open the contributing guide

Research direction

Start with IndexSearcher’s clause-count visitor and the visit methods in IndexOrDocValuesQuery and SortedNumericDocValuesRangeQuery, then trace how BooleanQuery contributes its children. Use the provided testIndexOrDocValues scenario as regression coverage; done means both rewrites complete below maxClauseCount without TooManyNestedClauses.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.