lance-format / lance-format/lance

bug: row_range panics where partition_size returns 0 for an absent partition

Open
#9,012 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

IvfModel::row_range and IvfModel::partition_size answer the same question about the same input and disagree: for a partition the model does not carry, one panics and the other returns 0.

pub fn row_range(&self, partition: usize) -> Range<usize> {
    let start = self.offsets[partition];          // storage.rs:144, panics
    let end = start + self.lengths[partition] as usize;
    start..end
}

pub fn partition_size(&self, part: usize) -> usize {
    self.lengths.get(part).copied().unwrap_or_default() as usize   // storage.rs:94, returns 0
}

num_partitions() adds to the confusion: it prefers centroids.len() and only falls back to offsets.len() (storage.rs:86-91), so the count a caller iterates over is not necessarily the one row_range can index.

I could not construct the panic through the public API, and this is worth saying plainly rather than filing it as a live bug. User-supplied partition ids reach load_partition, which rejects out-of-range ids with a clean error, and search-path ids come from find_partitions over the centroids so they are below nlist. The three callers that index without a guard would need a written model whose num_partitions() exceeds offsets.len(), and neither the builder nor the distributed merger can emit one.

The one seam I could not rule out: nothing checks the centroid count against lengths.len() on write or on load, and rust/lance/src/index/vector/ivf.rs:2690 grafts centroids from a different file than the one supplying the lengths, so a mismatched shard set there would reach the panic. I could not produce that from a normal write path.

So this is a contract inconsistency between two neighbouring methods rather than a reachable failure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with IvfModel::row_range, partition_size, and num_partitions in storage.rs:86-91, 94, and 144, then inspect the centroid grafting path at rust/lance/src/index/vector/ivf.rs:2690. Determine the intended contract for absent or mismatched partitions and add coverage for the relevant model shape; done means the neighboring methods behave consistently without an unhandled panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.