Can we store only quantized vectors to reduce disk footprint?
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
In light of optimizing disk usage for KNN vector searches in Lucene, I propose considering a new KnnVectorsFormat class in Lucene that handles only quantized vectors, eliminating the need to store original float32 vectors. This approach could significantly reduce disk usage, with potential reductions similar to the memory efficiency seen in int8 quantization scenarios, where usage can drop to about 25%. This figure is illustrative, emphasizing that actual savings could vary with different quantization methods and storage configurations.
I seek community feedback on:
- The technical feasibility of this new storage model.
- Potential impacts on search accuracy and performance.
Your insights will help determine the viability of this approach for enhancing Lucene's vector search capabilities.
Contributor guide
Research direction
Start by reviewing Lucene's existing KNN vector storage and quantization implementations, then assess the proposed KnnVectorsFormat class as the entry point. Determine whether storing only quantized vectors is technically feasible and document the expected effects on search accuracy, performance, and disk usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100