lance-format / lance-format/lance

bug: the kmeans balance factor is NaN when nothing is assigned, and f32::min hides it

Open Beginner friendly
#9,406 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

compute_cluster_sizes in rust/lance-index/src/vector/kmeans.rs returns the adjusted balance factor by dividing the largest cluster's loss by that cluster's size:

(radius[max_cluster_id] - losses[max_cluster_id] as f32 / cluster_sizes[max_cluster_id] as f32)
    / membership.len() as f32

When every membership is None no cluster has any rows, so the divisor is zero and the result is NaN. That is reachable: compute_membership_and_loss leaves a row unassigned when its vector is not finite, so an all-NaN training batch assigns nothing. KMeans::new_with_params is public, and the index build path filters non-finite vectors earlier, so this shows up through direct kmeans use.

The NaN then reaches the training loop:

let balance_factor = adjusted_balance_factor.min(params.balance_factor);

f32::min returns the non-NaN operand, so the NaN is swallowed and the loop silently continues with the configured factor. Nothing is corrupted today, but which factor the next iteration uses is decided by f32::min's NaN handling rather than by the code, and the value also feeds the balance loss that the convergence check compares.

Expected behavior

Say "no balancing" explicitly when nothing was assigned, instead of producing a NaN and relying on the caller's min to drop it.

Lance version

13.0.0-beta.4 (main)

Language binding

Rust

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance-index/src/vector/kmeans.rs at compute_cluster_sizes and trace the adjusted balance factor into the training loop in KMeans::new_with_params. Exercise the all-non-finite training batch where every membership is None, then verify the no-assignment case produces an explicit no-balancing result without NaN affecting the balance loss or convergence check.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.