apache / apache/lucene

Can we add configuration on dropping raw vectors from quantized formats after some period of time?

Open
#13,251 7 comments 0 reactions 0 assignees View on GitHub
type:enhancement vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

Tangentially related to: https://github.com/apache/lucene/issues/13158

But, I have observed, that as the corpus reaches a fairly large size, the actual quantiles aren't changing much at all during segment merges. This is tricky to fully measure and make a promise about lossiness (users can always just start throwing garbage that shakes up the whole world). But if the data isn't a "bad actor", quantiles and quantization buckets become fairly stable over time.

Maybe we should add a configuration option, or a new codec, or some way to drop the raw floating point vectors. The 4x reduction in disk usage would be really nice for many use-cases.

I am not 100% sure how this would look (a threshold provided by the user, or we just do it based on internal statistics).

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. First investigate quantized vector handling during segment merges and compare the configuration, codec, and internal-statistics alternatives. Done should be defined as a clear way to drop raw floating-point vectors after a stability threshold while documenting the resulting lossiness.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.