lance-format / lance-format/lance

bug: IvfTransformer::new_flat stores the unconverted metric for cosine

Open Beginner friendly
#9,398 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

Every IvfTransformer constructor in rust/lance-index/src/vector/ivf.rs handles cosine the same way: push a NormalizeTransformer and continue with L2. new_flat does that for the transform chain but then stores the original metric on the transformer:

let dt = if distance_type == DistanceType::Cosine {
    transforms.push(Arc::new(NormalizeTransformer::new(vector_column)));
    MetricType::L2
} else {
    distance_type
};
// ... PartitionTransformer::new(centroids.clone(), dt, vector_column) ...
Self::new(centroids, distance_type, transforms)

The sibling constructors pass the converted metric (with_pq and the one directly above it both do). The stored metric is what IvfTransformer::find_partitions, compute_partitions, and compute_residual measure with, so on a cosine flat index those three disagree with the chain: the chain normalized the vectors and assigned partitions with L2, while the transformer's own methods answer in cosine. For unit vectors the ranking is the same, so partition choice does not change, but the distances come back on a different scale than the ones the chain wrote into __ivf_centroid_dist.

Expected behavior

new_flat should store the converted metric like its siblings, so the transformer measures the way its own chain assigned.

Lance version

13.0.0-beta.4 (main)

Language binding

Rust

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance-index/src/vector/ivf.rs and compare IvfTransformer::new_flat with the sibling constructors, especially how they convert and store the metric. Check find_partitions, compute_partitions, and compute_residual to confirm they use the stored metric. Done means cosine new_flat uses the converted L2 metric consistently with its transform chain.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.