MultiFieldQueryParser: default operator is not honored between term positions
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
[Edit corresponding pull request : #16446 ]
## Disclaimer
Exactly like I already did for bug #16441, I found this bug in our codebase and manually wrote the unit tests covering the various cases. I then used Claude AI to help identify the root cause and draft this report. Finally, I verified both the reported behavior and the proposed patch against Lucene's main branch (running all tests) before submitting this report and corresponding merge request.
I could not find any policy in the [Contributing to Lucene Guide](https://github.com/apache/lucene/blob/main/CONTRIBUTING.md) preventing me from going this way.
My apologies if this is not authorized.
## Summary
`MultiFieldQueryParser`'s javadoc states that under `AND_OPERATOR`:
> "the result will be: `+(title:term1 body:term1) +(title:term2 body:term2)`... In other words, all the query's terms must appear, but it doesn't matter in what fields they appear."
This guarantee only holds when each term reaches the parser as a **separate grammar-level token** (i.e. real whitespace between them, handled by `QueryParser.jj`'s own `Clause()`/`addClause()`/`MultiTerm()` machinery term-by-term). It does not hold when several terms are produced by the **analyzer alone** from a *single* piece of query text passed to one `getFieldQuery(null, text, quoted)` call — for example an escaped separator with no real whitespace, word decompounding, CJK segmentation, or synonym expansion. In that case, the terms are always combined with `Occur.SHOULD`, regardless of `setDefaultOperator(...)`.
## Reproduction
```java
Analyzer analyzer = new ClassicAnalyzer(); // splits on `:`, unlike StandardAnalyzer
MultiFieldQueryParser parser =
new MultiFieldQueryParser(new String[] {"field1", "field2"}, analyzer);
parser.setDefaultOperator(QueryParser.Operator.AND);
Query q = parser.parse(QueryParser.escape("foo:bar"));
System.out.println(q.toString());
```
**Expected:**
```
+(field1:foo field2:foo) +(field1:bar field2:bar)
```
(both terms required, each as a disjunction across fields — matching the class's documented AND semantics)
**Actual:**
```
(field1:foo field2:foo) (field1:bar field2:bar)
```
(no `+` at all — both term positions are optional, i.e. the query behaves as OR)
## Root cause
Inside `MultiFieldQueryParser#getFieldQuery(String field, String queryText, boolean quoted)`, when a per-field analysis yields more than one token (`maxTerms > 1`), the method groups the fields for each term position into its own `SHOULD`-combined `BooleanQuery` (correct: fields are alternatives for a given term). It then combines these **per-position groups** using:
```java
protected Query getMultiFieldQuery(List queries) throws ParseException {
BooleanQuery.Builder query = newBooleanQuery();
for (Query sub : queries) {
query.add(sub, BooleanClause.Occur.SHOULD); // always SHOULD
}
return query.build();
}
```
`getMultiFieldQuery` is also the method used to combine *fields* for a single term (where `SHOULD` is always correct, since fields are genuine alternatives). It is being reused here to combine *term positions* as well, where the semantics should instead depend on the default operator. There is no operator-aware step anywhere in this code path for the "single call, multiple analyzed terms" case — the class relies entirely on the surrounding grammar (`Clause()`/`addClause()`) to apply the operator between terms, which only works when the terms are separate grammar tokens to begin with.
### Version and environment details
Affected version: main and 10.4.0
Component: lucene-queryparser (classic package)
Affected classes:
`MultiFieldQueryParser#getFieldQuery(String, String, boolean)` / `getMultiFieldQuery`
Contributor guide
Research direction
Start in MultiFieldQueryParser#getFieldQuery and getMultiFieldQuery, then compare the QueryParser.jj Clause(), addClause(), and MultiTerm() flow described in the report. Reproduce the ClassicAnalyzer example with AND_OPERATOR; done means analyzed term positions honor the default operator while fields remain alternatives.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100