Data-blind scalar quantization
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
Add options for data-blind scalar quantization. Data blind quantization will allow users to drop the raw float vectors which will be a significant reduction in storage in all cases (~4x or more improvement) but limits flexibility (no re-quantization of input).
There are 3 parts to this change:
- [ ] Allow users to disable centering and drop the raw float vectors. This is usable at high bit rates (4-8 bits/dim) where users may not be interested in re-ranking the results.
- [ ] Implement random rotation of vectors and queries. This generates a more favorable value distribution for quantization and should help reclaim some of the accuracy losses in the first step.
- [ ] Implement residual vector quantization. After quantizing each vector `v`, quantize the residual `v - dequantize(quantize(v))` and write it to separate storage. This vector can be used to boost accuracy during reranking/rescoring or vector decoding while still allowing fast/cheap representations for graph traversal. This makes data blind quantization more palatable at low bit rates (1-2 bits/dim).
Contributor guide
Research direction
Begin by tracing Lucene's existing vector quantization path and separating the three requirements in the issue: dropping raw vectors, random rotation, and residual quantization. Confirm the storage, reranking, decoding, and accuracy expectations before implementation; done means each listed capability is supported and validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100