lance-format / lance-format/lance
bug: the kmeans balance factor is NaN when nothing is assigned, and f32::min hides it
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
compute_cluster_sizes in rust/lance-index/src/vector/kmeans.rs returns the adjusted balance factor by dividing the largest cluster's loss by that cluster's size:
(radius[max_cluster_id] - losses[max_cluster_id] as f32 / cluster_sizes[max_cluster_id] as f32)
/ membership.len() as f32
When every membership is None no cluster has any rows, so the divisor is zero and the result is NaN. That is reachable: compute_membership_and_loss leaves a row unassigned when its vector is not finite, so an all-NaN training batch assigns nothing. KMeans::new_with_params is public, and the index build path filters non-finite vectors earlier, so this shows up through direct kmeans use.
The NaN then reaches the training loop:
let balance_factor = adjusted_balance_factor.min(params.balance_factor);
f32::min returns the non-NaN operand, so the NaN is swallowed and the loop silently continues with the configured factor. Nothing is corrupted today, but which factor the next iteration uses is decided by f32::min's NaN handling rather than by the code, and the value also feeds the balance loss that the convergence check compares.
Expected behavior
Say "no balancing" explicitly when nothing was assigned, instead of producing a NaN and relying on the caller's min to drop it.
Lance version
13.0.0-beta.4 (main)
Language binding
Rust
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance-index/src/vector/kmeans.rs at compute_cluster_sizes and trace the adjusted balance factor into the training loop in KMeans::new_with_params. Exercise the all-non-finite training batch where every membership is None, then verify the no-assignment case produces an explicit no-balancing result without NaN affecting the balance loss or convergence check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100