apache / apache/lucene

Use run-length encoding when writing docIds in BKD tree [LUCENE-9178]

Open
#10,218 1 comment 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

I think we can easily check if it make sense to write docIds using length compression in the BKD tree. This can probably save some space in the case of Muti value documents, e.g LatLonShape and XYShape.

---
Migrated from [LUCENE-9178](https://issues.apache.org/jira/browse/LUCENE-9178) by Ignacio Vera (@iverase)

Contributor guide

Open the contributing guide

Research direction

Start by locating the BKD tree code that writes docIds, then inspect how it is used for multi-value documents such as LatLonShape and XYShape. Determine whether run-length encoding is beneficial for those cases; done means the approach is implemented with evidence of space savings and no regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.