apache / apache/lucene

Can we store only quantized vectors to reduce disk footprint?

Open
#14,007 14 comments 0 reactions 0 assignees View on GitHub
type:enhancement vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

In light of optimizing disk usage for KNN vector searches in Lucene, I propose considering a new KnnVectorsFormat class in Lucene that handles only quantized vectors, eliminating the need to store original float32 vectors. This approach could significantly reduce disk usage, with potential reductions similar to the memory efficiency seen in int8 quantization scenarios, where usage can drop to about 25%. This figure is illustrative, emphasizing that actual savings could vary with different quantization methods and storage configurations.

I seek community feedback on:

- The technical feasibility of this new storage model.
- Potential impacts on search accuracy and performance.

Your insights will help determine the viability of this approach for enhancing Lucene's vector search capabilities.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Lucene's existing KNN vector storage and quantization implementations, then assess the proposed KnnVectorsFormat class as the entry point. Determine whether storing only quantized vectors is technically feasible and document the expected effects on search accuracy, performance, and disk usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.