lance-format / lance-format/lance
bug: KMeans panics on cosine over a float column instead of rejecting it
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
lance.util.KMeans advertises "cosine" and panics on it. KMeans(k, "cosine").fit(float32_data) aborts the process instead of raising.
The dispatch in KMeans::new_with_params matches a float column against any distance type (rust/lance-index/src/vector/kmeans.rs, the (DataType::Float32, _) arms in both the hierarchical and the flat block), so cosine and hamming reach the float algorithm, whose membership pass has arms for L2 and Dot only:
_ => {
panic!(
"KMeans::find_partitions: {} is not supported",
distance_type
);
}
The same holds for assignment: a model built with the public KMeans::with_centroids and a cosine metric panics in compute_membership_and_distances, which also dispatches on (value_type, centroid_type, _).
Reproduced on 36bd4e27a: a cosine model over an 8-dimensional f32 array panics at rust/lance-index/src/vector/kmeans.rs:511 with KMeans::find_partitions: cosine is not supported. Both dispatches already have a catch-all arm that returns a descriptive error, so the panic is only reached because the float arms claim the combination.
There is one exception, and it is why cosine looks supported: when the centroid set is large enough that may_train_index builds an HNSW over it, assignment goes through the index and never reads the distance type, so training completes. So the same API panics at small k and returns a model at large k.
Two documentation errors point the same way. The module comment says cosine is supported by normalizing the vectors each iteration, and new_with_params repeats it. That implementation existed once, added in #1723 and removed in #2015, and the comment was carried forward. Callers normalize the input themselves and train with l2 (rust/lance/src/index/vector/ivf.rs does this before every training call, and asserts non-cosine at one site).
python/python/benchmarks/test_kmeans.py is the one in-tree caller that passes cosine over float. It sits above the HNSW threshold, which is why it does not panic.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance-index/src/vector/kmeans.rs, especially the float dispatch arms in KMeans::new_with_params and compute_membership_and_distances, then inspect python/python/benchmarks/test_kmeans.py and the cosine-related comments. Done means unsupported cosine and hamming combinations return the existing descriptive error consistently instead of reaching the float algorithm's panic paths, with the stale support documentation corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100