bug: weighted k-means pads the partition count with duplicate clusters, leaving them permanently empty

Description

train_weighted_hierarchical_f32_kmeans in rust/lance/src/index/vector/ivf.rs is the trainer the streaming coreset path uses (num_partitions > 256 with a streaming sample rate). When it forms fewer clusters than target_k, it pads the list by cloning the heaviest cluster under fresh ids until the count matches:

while clusters.len() < target_k {
    let duplicate = clusters.iter().max_by(/* by weight */)…;
    clusters.push(WeightedCluster { id: next_cluster_id, ..duplicate });
    next_cluster_id += 1;
}

A padded centroid is a bit-identical clone of the one it came from, and assignment picks the nearest centroid with a strict < argmin — the incumbent wins a tie — so no vector is ever assigned to a duplicate. The refinement passes preserve it too: a cluster that receives no member keeps its previous centroid, so a zero-weight duplicate stays an exact duplicate.

The result is an index whose num_partitions counts partitions that can never hold a row. That is not only cosmetic: a query near the duplicated centroid burns one of its nprobes on an empty partition, because find_partitions sorts all centroid distances and takes the first nprobes, and the duplicate ties at exactly the same distance as the original. Recall drops with nothing reported.

The flat hierarchical trainer already rejects this request — rust/lance-index/src/vector/kmeans.rs:1493, "Cannot create {target_k} IVF partitions: k-means could only form {n} non-empty clusters…". So the same user request errors on one path and silently produces unreachable partitions on the other.

Steps to reproduce

Build an IVF index over data with many near-duplicate vectors, with enough partitions to take the streaming coreset trainer:

ds.create_index("vector", "IVF_PQ", num_partitions=512, ...)
# with a streaming sample rate set, and data whose coreset rows cannot be
# split into 512 non-empty clusters

The build succeeds. The resulting IvfModel.centroids contains duplicate rows whose lengths[p] is 0.

Expected behavior

The same descriptive error the flat trainer returns, naming how many clusters could be formed and suggesting a smaller num_partitions.

Lance version

13.0.0-beta.4 (main)

Language binding

Python, Rust

Read train_weighted_hierarchical_f32_kmeans in rust/lance/src/index/vector/ivf.rs and compare its undersized-cluster handling with rust/lance-index/src/vector/kmeans.rs:1493. Reproduce with the streaming coreset path and many near-duplicate vectors; done means the request returns an error naming the formed cluster count and suggesting a smaller num_partitions instead of creating empty duplicate partitions.