apache / apache/lucene

Re-evaluate different ways to encode postings [LUCENE-10672]

Open
#11,707 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

In Lucene 4, we moved to FOR to encode postings because it woud give better throughput compared to VInts that we had been using until then. This was a time when Lucene would often need to evaluate entire postings lists, and optimizations like BS1 were very important for good performance.

Nowadays, Lucene performs more dynamic pruning and it's less frequent that Lucene needs to evaluate all hits that match a query. So the performance of `nextDoc()` has become a bit less relevant while the performance of `advance(target)` has become more relevant.

I wonder if we should re-evaluate other ways to encode postings that are theoretically better at skipping, such as Elias-Fano coding, since they support skipping directly on the encoded representation instead of requiring decoding a full block of integers where only a couple of them would be relevant.

---
Migrated from [LUCENE-10672](https://issues.apache.org/jira/browse/LUCENE-10672) by Adrien Grand (@jpountz)

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating Lucene's postings encoders and the benchmarks or tests covering nextDoc() and advance(target), then compare FOR and VInts with a candidate such as Elias-Fano. Done means a decided encoding direction supported by representative performance results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.