apache / apache/lucene

Reconsider the best way to encode postings now that we can skip non-competitive hits [LUCENE-8760]

Open
#9,805 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

The fact that we now skip non competitive hits has some implications to our postings:
 - we are now more likely to call advance vs. nextDoc
 - we are less likely to read term frequency for a given doc, since we only do that if the maximum score reported by impacts is competitive
- we are less likely to read positions for a given doc, since exact phrase queries first check the maximum score that would be obtained with a phrase freq equal to the minimum of all term freqs

It might be a good opportunity to re-explore the best way to encode postings.

---
Migrated from [LUCENE-8760](https://issues.apache.org/jira/browse/LUCENE-8760) by Adrien Grand (@jpountz)

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reviewing Lucene's postings encodings and the advance/nextDoc, term-frequency, positions, and impacts paths described in the issue. Done would require a decided encoding approach and evidence that it suits the changed access patterns.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.