Add nightly test that calculates recall for vector similarity spaces
- Dominant language
- Java
- Stars
- 3.6k
- Forks
- 1.4k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 88
Description
### Description
We should have a nightly test that verifies expected recall for different vector spaces.
While in the past we have leaned on lucene nightly benchmarks to detect different vectors being returned, I am thinking it would be better to have an actual functional nightly test in Lucene itself.
Two ways to approach this:
I am not 100% sure we can utilize random data, but we may be able to. But we should have many vectors and test building all the supported vector formats and ensure recall is within some acceptable range.
Have static vectors associated with some "golden recall" to ensure it never changes and if it is, we must recalculate it.
It would be great if these static vectors are not unit vectors so that we adequately can test all our vector spaces.
Contributor guide
Research direction
Start by reviewing the existing Lucene nightly benchmarks and the supported vector formats. Decide whether random or static non-unit vectors will provide the recall baseline, and define an acceptable recall range. Done means a functional nightly test calculates recall across the supported vector spaces and detects unexpected changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- search, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100