lance-format / lance-format/lance

bug: a malformed precomputed partitions file panics the index build

Open
#9,341 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

load_precomputed_partitions in rust/lance-index/src/vector/ivf/builder.rs reads the row_id and partition columns out of a user-supplied file with .expect plus an unchecked as_primitive:

let row_ids: &UInt64Array = batch
    .column_by_name("row_id")
    .expect("malformed partition file: missing row_id column")
    .as_primitive();

A file that is missing either column, or that has the right names with the wrong physical type, panics in the middle of the build instead of returning an error. The file is the user's: LanceDataset.create_index(..., precomputed_partition_dataset=<uri>) checks only that the dataset has a single fragment and a single data file, then hands the URI to the index builder, which opens it and scans whatever schema it finds.

Nulls are a quieter version of the same problem. Both columns are read through their values buffer, which ignores validity, and the writer that produces these files marks the fields nullable. A null partition is therefore read as whatever the buffer happens to hold at that slot, so the row is assigned to an arbitrary partition and the index is built wrong with no error at all.

Steps to reproduce
import lance, pyarrow as pa

# The partitions file the user hands to create_index. row_id is Int64 here,
# which is what a naive pa.array([...]) of Python ints produces.
parts = pa.table({
    "row_id": pa.array([0, 1, 2], pa.int64()),
    "partition": pa.array([0, 1, 0], pa.uint32()),
})
lance.write_dataset(parts, "parts.lance")

ds = lance.dataset("data.lance")
ds.create_index("vector", "IVF_PQ", num_partitions=2,
                precomputed_partition_dataset="parts.lance")
# panics: malformed partition file: missing row_id column
Expected behavior

An invalid_input error naming the column, the type expected and what the file has instead. Same for a file that carries nulls in either column.

Lance version

13.0.0-beta.4 (main)

Language binding

Python, Rust

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance-index/src/vector/ivf/builder.rs at load_precomputed_partitions, then reproduce the issue through LanceDataset.create_index(..., precomputed_partition_dataset=...) using the Python example. Trace schema and null handling for row_id and partition. Done means malformed or nullable columns return invalid_input errors naming the problem instead of panicking or silently building an incorrect index.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.