apache / apache/lucene

Numeric DV block compression ignores the gcd when computing the number of bits required [LUCENE-9899]

Open
#10,938 4 comments 0 reactions 0 assignees View on GitHub
legacy-jira-priority:Major type:enhancement
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

When numeric doc values are splitted per block we compute the number of bits per value [from the minimum and maximum value present in the block](https://github.com/apache/lucene/blob/d5d6dc079395c47cd6d12dcce3bcfdd2c7d9dc63/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesConsumer.java#L390). However, the greatest common divisor is not taken into account so the number is overvalued for cases where it is greater than 1.

---
Migrated from [LUCENE-9899](https://issues.apache.org/jira/browse/LUCENE-9899) by Jim Ferenczi (@jimczi), updated Apr 01 2021
Attachments: [LUCENE-9899.patch](https://apache.github.io/lucene-jira-archive/attachments/LUCENE-9899/LUCENE-9899.patch)

Contributor guide

Open the contributing guide

Research direction

Start in lucene/core/src/main/java/org/apache/lucene/codecs/lucene90/Lucene90DocValuesConsumer.java at the linked computation around line 390. Trace how the block's minimum and maximum values determine the required bits, then inspect the issue's attached LUCENE-9899.patch for context. Done means numeric doc-value blocks account for their greatest common divisor when calculating the bit width.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.