lance-format / lance-format/lance

bug: an out-of-range precomputed partition id panics the IVF_PQ build

Open
#9,003 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

create_index for IVF_PQ with precomputed_partition_dataset panics when the supplied file names a partition id beyond the centroid count. It is a slice panic, not an error, so from Python it comes out as a PanicException.

range start index 10 out of range for slice of length 4

do_compute_residual (rust/lance-index/src/vector/residual.rs:87) indexes the centroid buffer with the id straight from the batch:

let part_id = part_ids[idx] as usize;
let c = &centroids_slice[part_id * dimension..(part_id + 1) * dimension];

Partition ids normally come from the transformer that just assigned them, so they are in range by construction. A precomputed partitions dataset is different: it is user data, and nothing between the file and this slice checks it. rust/lance/src/index/vector/builder.rs:867-901 writes the map's values into PART_ID_COLUMN and filters nulls only, and PartitionTransformer::transform then early-returns because the column already exists (rust/lance-index/src/vector/ivf/transform.rs:77-81), so the user's ids survive into the residual step.

It surfaces as a panic rather than an error because the transform runs in a spawned task whose handle is unwrapped at builder.rs:914.

Only IVF_PQ is affected, for two reasons. The shuffler already range-checks partition ids and reports them properly (rust/lance-index/src/vector/v3/shuffler.rs:600-606), but the transform chain runs before the shuffle, so for IVF_PQ the residual slice gets there first. And IVF_RQ builds its partition transformer with .with_distance(true) (rust/lance-index/src/vector/ivf.rs:299), which makes the early-return condition false when no centroid-distance column is present, so RQ recomputes the ids and silently discards the supplied ones.

In Python the parameter is create_index(..., precomputed_partition_dataset=uri) with index_type="IVF_PQ" and an L2 or cosine metric.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at do_compute_residual in rust/lance-index/src/vector/residual.rs:87, then trace the precomputed partition path through rust/lance/src/index/vector/builder.rs:867-901 and 914 and PartitionTransformer::transform in rust/lance-index/src/vector/ivf/transform.rs:77-81. Compare the IVF_PQ path with the shuffler check in rust/lance-index/src/vector/v3/shuffler.rs:600-606 and IVF_RQ setup in rust/lance-index/src/vector/ivf.rs:299. Done means an out-of-range precomputed partition id is reported as an error from Python instead of causing a PanicException.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.