apache / apache/lucene

Improve BKDWriter's DocIdsWriter to better encode decreasing sequences of doc IDs [LUCENE-10409]

Open
#11,445 3 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:task
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

@gf2121 recently improved DocIdsWriter for the case when doc IDs are dense and come in the same order as values via the CONTINUOUS_IDS and BITSET_IDS encodings.

We could do the same for the case when doc IDs come in the opposite order to values. This would be used whenever searching on a field that is used for index sorting in the descending order. This would be a frequent case for Elasticsearch users as we're planning on using index sorting more and more on time-based data with a descending sort on the timestamp as the last sort field.

---
Migrated from [LUCENE-10409](https://issues.apache.org/jira/browse/LUCENE-10409) by Adrien Grand (@jpountz), updated Feb 10 2022
Pull requests: https://github.com/apache/lucene/pull/666

Contributor guide

Open the contributing guide

Research direction

Start by reading BKDWriter's DocIdsWriter and the existing CONTINUOUS_IDS and BITSET_IDS encodings to understand how dense, forward-ordered doc IDs are handled. Compare that behavior with decreasing doc-ID sequences, then check the linked pull request and confirm that the descending case is encoded efficiently without regressing existing cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.