apache / apache/lucene

StandardQueryParser is over 100 times slower in v5 compared to v3 [LUCENE-7260]

Open
#8,315 10 comments 0 reactions 0 assignees View on GitHub
affects-version:5.4.1 affects-version:5.5.2 affects-version:6.3 legacy-jira-label:performance legacy-jira-priority:Major module:queryparser type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The following test code times parsing a large query.

```Java
import org.apache.lucene.analysis.KeywordAnalyzer;
//import org.apache.lucene.analysis.core.KeywordAnalyzer;
import org.apache.lucene.queryParser.standard.StandardQueryParser;
//import org.apache.lucene.queryparser.flexible.standard.StandardQueryParser;
import org.apache.lucene.search.BooleanQuery;

public class LargeQueryTest {
public static void main(String[] args) throws Exception {
BooleanQuery.setMaxClauseCount(50_000);
StringBuilder builder = new StringBuilder(50_000*10);
builder.append("id:( ");
boolean first = true;
for (int i = 0; i < 50_000; i++) {
if (first) {
first = false;
} else {
builder.append(" OR ");
}
builder.append(String.valueOf(i));
}
builder.append(" )");
String queryString = builder.toString();

StandardQueryParser parser2 = new StandardQueryParser(new KeywordAnalyzer());

for (int i = 0; i < 10; i++) {
long t0 = System.currentTimeMillis();
parser2.parse(queryString, "nope");
long t1 = System.currentTimeMillis();
System.out.println(t1-t0);
}
}
}
```

For Lucene 3.6.2, the timings settle down to 200\~300 with the fastest being 207.
For Lucene 5.4.1, the timings settle down to 20000\~30000 with the fastest being 22444.

So at some point, some change made the query parser 100 times slower. I would suspect that it has something to do with how the list of children is now handled. Every time someone gets the children, it copies the list. Every time someone sets the children, it walks through to detach parent references and then reattaches them all again.

If it were me, I would probably make these collections immutable so that I didn't have to defensively copy them.

---
Migrated from [LUCENE-7260](https://issues.apache.org/jira/browse/LUCENE-7260) by Trejkaz, 1 vote, updated Feb 16 2017
Environment:
```
Java 8u51
```

Contributor guide

Open the contributing guide

Research direction

Start with the LargeQueryTest reproduction and the StandardQueryParser.parse entry point, then compare parsing behavior between Lucene 3.6.2 and 5.4.1. Investigate the reported child-list copying and parent-reference handling, and use the 50,000-clause query timings to verify that the regression is addressed without changing parsing results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.