apache / apache/lucene

MultiFieldQueryParser: default operator is not honored between term positions

Open
#16,443 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

[Edit corresponding pull request : #16446 ]

## Disclaimer

Exactly like I already did for bug #16441, I found this bug in our codebase and manually wrote the unit tests covering the various cases. I then used Claude AI to help identify the root cause and draft this report. Finally, I verified both the reported behavior and the proposed patch against Lucene's main branch (running all tests) before submitting this report and corresponding merge request.

I could not find any policy in the [Contributing to Lucene Guide](https://github.com/apache/lucene/blob/main/CONTRIBUTING.md) preventing me from going this way.
My apologies if this is not authorized.

## Summary

`MultiFieldQueryParser`'s javadoc states that under `AND_OPERATOR`:

> "the result will be: `+(title:term1 body:term1) +(title:term2 body:term2)`... In other words, all the query's terms must appear, but it doesn't matter in what fields they appear."

This guarantee only holds when each term reaches the parser as a **separate grammar-level token** (i.e. real whitespace between them, handled by `QueryParser.jj`'s own `Clause()`/`addClause()`/`MultiTerm()` machinery term-by-term). It does not hold when several terms are produced by the **analyzer alone** from a *single* piece of query text passed to one `getFieldQuery(null, text, quoted)` call — for example an escaped separator with no real whitespace, word decompounding, CJK segmentation, or synonym expansion. In that case, the terms are always combined with `Occur.SHOULD`, regardless of `setDefaultOperator(...)`.

## Reproduction

```java
Analyzer analyzer = new ClassicAnalyzer(); // splits on `:`, unlike StandardAnalyzer
MultiFieldQueryParser parser =
new MultiFieldQueryParser(new String[] {"field1", "field2"}, analyzer);
parser.setDefaultOperator(QueryParser.Operator.AND);

Query q = parser.parse(QueryParser.escape("foo:bar"));
System.out.println(q.toString());
```
**Expected:**
```
+(field1:foo field2:foo) +(field1:bar field2:bar)
```
(both terms required, each as a disjunction across fields — matching the class's documented AND semantics)

**Actual:**
```
(field1:foo field2:foo) (field1:bar field2:bar)
```
(no `+` at all — both term positions are optional, i.e. the query behaves as OR)

## Root cause

Inside `MultiFieldQueryParser#getFieldQuery(String field, String queryText, boolean quoted)`, when a per-field analysis yields more than one token (`maxTerms > 1`), the method groups the fields for each term position into its own `SHOULD`-combined `BooleanQuery` (correct: fields are alternatives for a given term). It then combines these **per-position groups** using:

```java
protected Query getMultiFieldQuery(List queries) throws ParseException {
BooleanQuery.Builder query = newBooleanQuery();
for (Query sub : queries) {
query.add(sub, BooleanClause.Occur.SHOULD); // always SHOULD
}
return query.build();
}
```

`getMultiFieldQuery` is also the method used to combine *fields* for a single term (where `SHOULD` is always correct, since fields are genuine alternatives). It is being reused here to combine *term positions* as well, where the semantics should instead depend on the default operator. There is no operator-aware step anywhere in this code path for the "single call, multiple analyzed terms" case — the class relies entirely on the surrounding grammar (`Clause()`/`addClause()`) to apply the operator between terms, which only works when the terms are separate grammar tokens to begin with.

### Version and environment details

Affected version: main and 10.4.0
Component: lucene-queryparser (classic package)
Affected classes:
`MultiFieldQueryParser#getFieldQuery(String, String, boolean)` / `getMultiFieldQuery`

Contributor guide

Open the contributing guide

Research direction

Start in MultiFieldQueryParser#getFieldQuery and getMultiFieldQuery, then compare the QueryParser.jj Clause(), addClause(), and MultiTerm() flow described in the report. Reproduce the ClassicAnalyzer example with AND_OPERATOR; done means analyzed term positions honor the default operator while fields remain alternatives.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.