Give TermInSetQuery better advancing capabilities [LUCENE-7958]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
If a TermInSetQuery has more than 15 matching terms on a given segment, then we consume all postings lists into a bitset and return an iterator over this bitset as a scorer. I would like to change it so that we keep the 15 postings lists that have the largest document frequencies and consume all other (shorter) postings lists into a bitset. In the end we return a disjunction over the N longest postings lists and the bit set. This could help consume fewer doc ids if the TermInSetQuery is intersected with other queries, especially if the document frequencies of the terms it wraps have a zipfian distribution.
---
Migrated from [LUCENE-7958](https://issues.apache.org/jira/browse/LUCENE-7958) by Adrien Grand (@jpountz), 1 vote, updated Mar 19 2019
Attachments: [LUCENE-7958.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7958/LUCENE-7958.patch)
Contributor guide
Research direction
Start at TermInSetQuery, focusing on the segment path used when more than 15 terms match. Verify that the 15 postings lists with the largest document frequencies remain separate and that shorter lists are represented by a bitset. Done means the resulting disjunction supports more efficient advancing when intersected with other queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100