apache / apache/lucene

Data-blind scalar quantization

Open
#16,029 8 comments 1 reaction 0 assignees View on GitHub
type:enhancement vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Add options for data-blind scalar quantization. Data blind quantization will allow users to drop the raw float vectors which will be a significant reduction in storage in all cases (~4x or more improvement) but limits flexibility (no re-quantization of input).

There are 3 parts to this change:
- [ ] Allow users to disable centering and drop the raw float vectors. This is usable at high bit rates (4-8 bits/dim) where users may not be interested in re-ranking the results.
- [ ] Implement random rotation of vectors and queries. This generates a more favorable value distribution for quantization and should help reclaim some of the accuracy losses in the first step.
- [ ] Implement residual vector quantization. After quantizing each vector `v`, quantize the residual `v - dequantize(quantize(v))` and write it to separate storage. This vector can be used to boost accuracy during reranking/rescoring or vector decoding while still allowing fast/cheap representations for graph traversal. This makes data blind quantization more palatable at low bit rates (1-2 bits/dim).

Contributor guide

Open the contributing guide

Research direction

Begin by tracing Lucene's existing vector quantization path and separating the three requirements in the issue: dropping raw vectors, random rotation, and residual quantization. Confirm the storage, reranking, decoding, and accuracy expectations before implementation; done means each listed capability is supported and validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.