lance-format / lance-format/lance
bug: an out-of-range precomputed partition id panics the IVF_PQ build
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
create_index for IVF_PQ with precomputed_partition_dataset panics when the supplied file names a partition id beyond the centroid count. It is a slice panic, not an error, so from Python it comes out as a PanicException.
range start index 10 out of range for slice of length 4
do_compute_residual (rust/lance-index/src/vector/residual.rs:87) indexes the centroid buffer with the id straight from the batch:
let part_id = part_ids[idx] as usize;
let c = ¢roids_slice[part_id * dimension..(part_id + 1) * dimension];
Partition ids normally come from the transformer that just assigned them, so they are in range by construction. A precomputed partitions dataset is different: it is user data, and nothing between the file and this slice checks it. rust/lance/src/index/vector/builder.rs:867-901 writes the map's values into PART_ID_COLUMN and filters nulls only, and PartitionTransformer::transform then early-returns because the column already exists (rust/lance-index/src/vector/ivf/transform.rs:77-81), so the user's ids survive into the residual step.
It surfaces as a panic rather than an error because the transform runs in a spawned task whose handle is unwrapped at builder.rs:914.
Only IVF_PQ is affected, for two reasons. The shuffler already range-checks partition ids and reports them properly (rust/lance-index/src/vector/v3/shuffler.rs:600-606), but the transform chain runs before the shuffle, so for IVF_PQ the residual slice gets there first. And IVF_RQ builds its partition transformer with .with_distance(true) (rust/lance-index/src/vector/ivf.rs:299), which makes the early-return condition false when no centroid-distance column is present, so RQ recomputes the ids and silently discards the supplied ones.
In Python the parameter is create_index(..., precomputed_partition_dataset=uri) with index_type="IVF_PQ" and an L2 or cosine metric.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at do_compute_residual in rust/lance-index/src/vector/residual.rs:87, then trace the precomputed partition path through rust/lance/src/index/vector/builder.rs:867-901 and 914 and PartitionTransformer::transform in rust/lance-index/src/vector/ivf/transform.rs:77-81. Compare the IVF_PQ path with the shuffler check in rust/lance-index/src/vector/v3/shuffler.rs:600-606 and IVF_RQ setup in rust/lance-index/src/vector/ivf.rs:299. Done means an out-of-range precomputed partition id is reported as an error from Python instead of causing a PanicException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- databases, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100