apache / apache/lucene

Add large-scale test for kNN vectors

Open
#11,863 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.6k
Forks
1.4k
Avg merge
2d 11h
Merged PRs (30d)
88

Description

We recently had a regression where the kNN vectors format validation could fail on large segments. We didn't catch this in testing or nightly performance benchmarks because they didn't produce large enough segments to trigger the bug.

We should add a "monster" test similar to `Test4BBKDPoints` that indexes a large dataset and maybe does a force merge. When tests are labelled `@Monster` they won't run as part of the regular build and are allowed to take much longer.

Contributor guide

Open the contributing guide

Research direction

Locate the existing `Test4BBKDPoints` test and review how large datasets, force merges, and the `@Monster` label are handled. Add a comparable kNN vectors test that creates segments large enough to exercise the regression, then verify it runs as a monster test and catches the large-segment format-validation failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
search, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.