optimize spantermquery [LUCENE-2780]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
Looking at http://www.lucidimagination.com/search/document/c2c6f660ddde4f7f/dismaxqparserplugin_and_tokenization ,
I saw a user building DisjunctionMaxQuery / BooleanQuery with SpanTermQuerys.
I wonder if users know that doing this is much slower than just using TermQuery?
I agree it makes little sense to use SpanTermQuery if you arent going to use it inside a SpanNear etc,
but on the other hand, I think its a little non-intuitive that it wouldnt be just as fast in a case like this.
I could see this complicating queryparsing etc for users that want to sometimes use positions etc.
SpanTermQuery is the same as TermQuery, except tf is computed as (#of spans \* sloppyFreq(spanLength)
For this case, #ofspans = tf and spanLength for a single term is always 1.
Maybe we should optimize SpanTermQuery to return TermScorer, with just this special tf computation.
This would avoid reading positions for anyone that does this.
---
Migrated from [LUCENE-2780](https://issues.apache.org/jira/browse/LUCENE-2780) by Robert Muir (@rmuir), updated May 09 2016
Attachments: [LUCENE-2780.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-2780/LUCENE-2780.patch)
Contributor guide
Research direction
Start by reading SpanTermQuery, TermQuery, TermScorer, DisjunctionMaxQuery, and BooleanQuery, then inspect the attached LUCENE-2780.patch. Compare how SpanTermQuery scores a single term with TermQuery and determine the expected behavior when positions are not needed; done means the optimization preserves scoring semantics and avoids unnecessary position reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100