apache / apache/lucene

Add nightly test that calculates recall for vector similarity spaces

Open
#13,616 1 comment 0 reactions 0 assignees View on GitHub
type:task vector-based-search
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

### Description

We should have a nightly test that verifies expected recall for different vector spaces.

While in the past we have leaned on lucene nightly benchmarks to detect different vectors being returned, I am thinking it would be better to have an actual functional nightly test in Lucene itself.

Two ways to approach this:

I am not 100% sure we can utilize random data, but we may be able to. But we should have many vectors and test building all the supported vector formats and ensure recall is within some acceptable range.

Have static vectors associated with some "golden recall" to ensure it never changes and if it is, we must recalculate it.

It would be great if these static vectors are not unit vectors so that we adequately can test all our vector spaces.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing Lucene nightly benchmarks and the supported vector formats. Decide whether random or static non-unit vectors will provide the recall baseline, and define an acceptable recall range. Done means a functional nightly test calculates recall across the supported vector spaces and detects unexpected changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.