apache / apache/lucene

optimize spantermquery [LUCENE-2780]

Open
#3,854 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-fix-version:4.9 legacy-jira-fix-version:6.0 legacy-jira-priority:Major module:core/search type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Looking at http://www.lucidimagination.com/search/document/c2c6f660ddde4f7f/dismaxqparserplugin_and_tokenization ,
I saw a user building DisjunctionMaxQuery / BooleanQuery with SpanTermQuerys.

I wonder if users know that doing this is much slower than just using TermQuery?
I agree it makes little sense to use SpanTermQuery if you arent going to use it inside a SpanNear etc,
but on the other hand, I think its a little non-intuitive that it wouldnt be just as fast in a case like this.

I could see this complicating queryparsing etc for users that want to sometimes use positions etc.

SpanTermQuery is the same as TermQuery, except tf is computed as (#of spans \* sloppyFreq(spanLength)
For this case, #ofspans = tf and spanLength for a single term is always 1.

Maybe we should optimize SpanTermQuery to return TermScorer, with just this special tf computation.
This would avoid reading positions for anyone that does this.

---
Migrated from [LUCENE-2780](https://issues.apache.org/jira/browse/LUCENE-2780) by Robert Muir (@rmuir), updated May 09 2016
Attachments: [LUCENE-2780.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2780/LUCENE-2780.patch)

Contributor guide

Open the contributing guide

Research direction

Start by reading SpanTermQuery, TermQuery, TermScorer, DisjunctionMaxQuery, and BooleanQuery, then inspect the attached LUCENE-2780.patch. Compare how SpanTermQuery scores a single term with TermQuery and determine the expected behavior when positions are not needed; done means the optimization preserves scoring semantics and avoids unnecessary position reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.