apache / apache/lucene

Query parser should generate FieldValueFilter for pure wildcard terms to boost query performance [LUCENE-4386]

Open
#5,452 8 comments 0 reactions 0 assignees View on GitHub
affects-version:4.0-BETA legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-priority:Major module:core/queryparser type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In theory, a simple pure wildcard query (a single asterisk) is an inefficient way to select all documents that have any value in a field. Rather than users having to work around this issue by adding a separate boolean "has" field, it would be better to have the query parser directly generate the most efficient Lucene query for detecting all documents that have any value for a specified field. According to the discussion over on #5442, the FieldValueFilter is the proper solution.

Proposed solution:

QueryParserBase.getPrefixQuery could detect when the query is a pure wildcard (a single asterisk) and then generate a FieldValueFilter instead of a PrefixQuery. My understanding from #5442 is that the following would work:

```Java
new ConstantScoreQuery(new FieldValueFilter(fieldname, false))
```

Oh, and the check for whether "leading wildcard" is enabled would need to be bypassed for this case.

I still think it would be better to have PrefixQuery perform this optimization internally so that all apps would benefit, but this should be sufficient to address the main concern.

This improvement would improve the classic Lucene query parser and other query parsers based on it, including edismax. There might be other query parsers which won't see the impact of this change, but they can be updated separately.

How much performance benefit? Unknown, but supposedly significant. The goal is simply to have a simple pure wildcard be the obvious tool to select fields that have a value in a field.

---
Migrated from [LUCENE-4386](https://issues.apache.org/jira/browse/LUCENE-4386) by Jack Krupansky, 1 vote, updated May 09 2016

Contributor guide

Open the contributing guide

Research direction

Start at QueryParserBase.getPrefixQuery and inspect how FieldValueFilter, PrefixQuery, and ConstantScoreQuery are used in the classic query parser. Verify the pure single-asterisk case and its interaction with leading-wildcard checks; done means the parser produces the proposed field-value query for that case while other prefix queries retain their existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.