lance-format / lance-format/lance
bug: IVF_HNSW_PQ recall assertions fail intermittently because k-means init is unseeded
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
test_create_ivf_hnsw_pq_4bit and test_create_ivf_hnsw_pq_multivec fail their
assert_ge!(recall, 0.5) check intermittently, on main as well as on PRs.
Observed in the last day:
| where | test | recall |
|---|---|---|
main 91aee6f91, linux-build |
test_create_ivf_hnsw_pq_4bit::case_3_dot |
0.4 |
PR #8431, mac-build (stable) |
test_create_ivf_hnsw_pq_4bit::case_3_dot |
0.4 |
PR #8755, linux-arm |
test_create_ivf_hnsw_pq_multivec |
0.3 |
Each run was otherwise green (e.g. 3146 passed, 1 failed), and the failing test
differs by runner, so this is not a regression from any one change.
The input is already deterministic: make_seeded_vector_batch
(rust/lance/src/index/vector/ivf/v2.rs:2615) pins Seed::from(42). The variance
comes from index training instead — rust/lance-index/src/vector/kmeans.rs:930
reads
// TODO: use seed for Rng.
let mut rng = SmallRng::from_os_rng();
so centroid initialization differs every run. With K = 10 the recall metric
moves in steps of 0.1 and the threshold is 5 of 10 neighbours, so a slightly
worse set of centroids flips the assertion.
Threading a seed through KMeansParams for the test path would make these
deterministic, which the existing TODO already anticipates. Lowering the
threshold would also stop the failures but would weaken the check.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with rust/lance-index/src/vector/kmeans.rs:930 and inspect how KMeansParams reaches training for test_create_ivf_hnsw_pq_4bit and test_create_ivf_hnsw_pq_multivec. Compare this with the seeded input setup in rust/lance/src/index/vector/ivf/v2.rs:2615, then run both named tests repeatedly. Done means centroid initialization is deterministic for these tests and the existing recall assertions pass without lowering their threshold.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100