apache / apache/lucene

Refactoring multiterm analysis [LUCENE-5470]

Open
#6,533 7 comments 0 reactions 0 assignees View on GitHub
affects-version:4.7 legacy-jira-priority:Minor module:core/queryparser type:bug
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

There are currently three methods to analyze multiterms in Lucene and Solr:

1) QueryParserBase
2) AnalyzingQueryParser
3) TextField (Solr)

The code in QueryParserBase and in TextField do not consume the tokenstream if more than one token is generated by the analyzer. (Admittedly, thanks to the magic of MultitermAwareComponents in Solr, this type of exception probably never happens and the unconsumed stream problem is probably non-existent in Solr.)

I propose consolidating the multiterm analysis code into one place: QueryBuilder in Lucene core.

This is part of a refactoring that will also help reduce duplication of code with #6269.

---
Migrated from [LUCENE-5470](https://issues.apache.org/jira/browse/LUCENE-5470) by Tim Allison (@tballison), 1 vote, updated Apr 09 2014
Attachments: [LUCENE-5470_QBuilder.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5470/LUCENE-5470_QBuilder.patch), [LUCENE-5470_QPBase.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5470/LUCENE-5470_QPBase.patch), [LUCENE-5470.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-5470/LUCENE-5470.patch)

Contributor guide

Open the contributing guide

Research direction

Compare the multiterm analysis implementations in QueryParserBase, AnalyzingQueryParser, and Solr's TextField, then read the attached LUCENE-5470 patches and the QueryBuilder entry point. The intended result is a single QueryBuilder location for this analysis, including handling streams that produce multiple tokens and reducing duplication with issue #6269.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.