lance-format / lance-format/lance

bug: the Python binding drops shuffle_partition_batches

Open Beginner friendly
#9,387 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

prepare_vector_index_params in python/src/dataset.rs reads shuffle_partition_concurrency out of the kwargs dict and never reads shuffle_partition_batches, so the second one is dropped between Python and IvfBuildParams:

if let Some(n) = kwargs.get_item("shuffle_partition_concurrency")? {
    ivf_params.shuffle_partition_concurrency = n.extract()?
};

LanceDataset.create_index documents the parameter ("The number of batches, using the row group size of the dataset, to include in each shuffle partition. Default value is 10240"), puts it into the kwargs dict, and test_create_index_shuffle_params passes shuffle_partition_batches=1, which quietly built with 10240 the whole time, because IvfBuildParams::default() happens to carry the same number the docstring names.

The value is observable through the index's runtime hints. vector_index_details records lance.ivf.shuffle_partition_batches next to lance.ivf.shuffle_partition_concurrency, so a build that asked for 512 batches reports 10240 in describe_indices()[0].details, and apply_runtime_hints then replays that wrong number on a later optimize.

The legacy IVF_PQ and IVF_HNSW_PQ writers are what consume the value (build_partitions passes it to shuffle_dataset); the V3 builder shuffles by bytes and does not read it. So the fix restores the parameter for the writer that has always read it, and stops misreporting it for every writer.

Expected behavior

Parse shuffle_partition_batches where its sibling is parsed, so the value the caller passes reaches IvfBuildParams and the recorded hint matches the build.

Lance version

13.0.0-beta.4 (main)

Language binding

Python

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in python/src/dataset.rs at prepare_vector_index_params, where shuffle_partition_concurrency is parsed, and inspect test_create_index_shuffle_params. Verify that shuffle_partition_batches reaches IvfBuildParams and that the recorded lance.ivf.shuffle_partition_batches value in describe_indices()[0].details matches the requested build value, including when runtime hints are replayed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.