Improve BKDWriter's DocIdsWriter to better encode decreasing sequences of doc IDs [LUCENE-10409]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
@gf2121 recently improved DocIdsWriter for the case when doc IDs are dense and come in the same order as values via the CONTINUOUS_IDS and BITSET_IDS encodings.
We could do the same for the case when doc IDs come in the opposite order to values. This would be used whenever searching on a field that is used for index sorting in the descending order. This would be a frequent case for Elasticsearch users as we're planning on using index sorting more and more on time-based data with a descending sort on the timestamp as the last sort field.
---
Migrated from [LUCENE-10409](https://issues.apache.org/jira/browse/LUCENE-10409) by Adrien Grand (@jpountz), updated Feb 10 2022
Pull requests: https://github.com/apache/lucene/pull/666
Contributor guide
Research direction
Start by reading BKDWriter's DocIdsWriter and the existing CONTINUOUS_IDS and BITSET_IDS encodings to understand how dense, forward-ordered doc IDs are handled. Compare that behavior with decreasing doc-ID sequences, then check the linked pull request and confirm that the descending case is encoded efficiently without regressing existing cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100