apache / apache/lucene

Bound the number of impacts per block [LUCENE-9156]

Open
#10,196 2 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

Today we write all competitive impacts in skip lists. Maybe we should be more cautious: in the worst-case scenario, there could be as many impacts as postings in a block and this would not only waste space but also slow down queries due to the time it would take to decode all these impacts.

One idea would be to merge all impacts pairwise, taking the maximum term frequency and the minimum norm, until their number is under some hard-coded limit, e.g. 8.

---
Migrated from [LUCENE-9156](https://issues.apache.org/jira/browse/LUCENE-9156) by Adrien Grand (@jpountz), updated Jan 22 2020

Contributor guide

Open the contributing guide

Research direction

Start with the migrated LUCENE-9156 proposal and the code that writes competitive impacts in skip lists. Determine how a hard-coded impact bound and pairwise merging would affect space use and query-time decoding. Done means the number of impacts per block is bounded without losing the stated maximum term frequency and minimum norm behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.