apache / apache/lucene

Give TermInSetQuery better advancing capabilities [LUCENE-7958]

Open
#9,007 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

If a TermInSetQuery has more than 15 matching terms on a given segment, then we consume all postings lists into a bitset and return an iterator over this bitset as a scorer. I would like to change it so that we keep the 15 postings lists that have the largest document frequencies and consume all other (shorter) postings lists into a bitset. In the end we return a disjunction over the N longest postings lists and the bit set. This could help consume fewer doc ids if the TermInSetQuery is intersected with other queries, especially if the document frequencies of the terms it wraps have a zipfian distribution.

---
Migrated from [LUCENE-7958](https://issues.apache.org/jira/browse/LUCENE-7958) by Adrien Grand (@jpountz), 1 vote, updated Mar 19 2019
Attachments: [LUCENE-7958.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-7958/LUCENE-7958.patch)

Contributor guide

Open the contributing guide

Research direction

Start at TermInSetQuery, focusing on the segment path used when more than 15 terms match. Verify that the 15 postings lists with the largest document frequencies remain separate and that shorter lists are represented by a bitset. Done means the resulting disjunction supports more efficient advancing when intersected with other queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.