Compress BKD trees by encoding the difference between two dimensions [LUCENE-8705]
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
When serializing BKD trees to disk, for each block we look at the common prefix for each dimension in isolation and only encode those common prefixes once for the entire block. Now that we have range fields and shapes so that several dimensions are storing related data, we might occasionally have longer common prefixes when comparing with values in other dimensions. For instance when indexing narrow ranges in a range field, we might get better compression on the second dimension by encoding suffixes that differ with the first dimension. This is also an obvious win if we are indexing lines or points as shapes, since we have dimensions that record exactly the same values in that case.
---
Migrated from [LUCENE-8705](https://issues.apache.org/jira/browse/LUCENE-8705) by Adrien Grand (@jpountz)
Contributor guide
Research direction
Start with the issue description and trace Lucene's BKD-tree serialization path; no file, test, or entry point is named. The work is done when related dimensions can share common-prefix encoding while preserving correct BKD data and demonstrating improved compression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100