apache / apache/lucene

Compress BKD trees by encoding the difference between two dimensions [LUCENE-8705]

Open
#9,751 0 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Minor type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When serializing BKD trees to disk, for each block we look at the common prefix for each dimension in isolation and only encode those common prefixes once for the entire block. Now that we have range fields and shapes so that several dimensions are storing related data, we might occasionally have longer common prefixes when comparing with values in other dimensions. For instance when indexing narrow ranges in a range field, we might get better compression on the second dimension by encoding suffixes that differ with the first dimension. This is also an obvious win if we are indexing lines or points as shapes, since we have dimensions that record exactly the same values in that case.

---
Migrated from [LUCENE-8705](https://issues.apache.org/jira/browse/LUCENE-8705) by Adrien Grand (@jpountz)

Contributor guide

Open the contributing guide

Research direction

Start with the issue description and trace Lucene's BKD-tree serialization path; no file, test, or entry point is named. The work is done when related dimensions can share common-prefix encoding while preserving correct BKD data and demonstrating improved compression.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.