lance-format / lance-format/lance

bug: create_index silently ignores num_partitions when ivf_centroids_file is given

Open
#9,008 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

create_index silently ignores num_partitions when ivf_centroids_file is also given. The file's cluster count wins, with no error and no warning:

ds.create_index(
    ["vector"], index_type="IVF_PQ",
    ivf_centroids_file="centroids.npy",   # 4 clusters
    num_partitions=8,
    num_sub_vectors=8,
)

builds a 4-partition index.

The branch loads the file and then overwrites the caller's value (python/python/lance/dataset.py:4048):

ivf_centroids = np.load(f)
num_partitions = ivf_centroids.shape[0]

The only mismatch guard in this function is on the numpy-array path (:4108-4111), and by the time it runs num_partitions has already been replaced by the file's count, so it compares the value against itself and can never fire for the file path.

Nothing documents which one is supposed to win. num_partitions has a docstring entry that marks it deprecated in favor of target_partition_size (:4268); ivf_centroids_file has no docstring entry anywhere. So a caller who passes both has no way to know that one of the two arguments was discarded.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in python/python/lance/dataset.py at create_index around lines 4048, 4108-4111, and 4268. Compare the ivf_centroids_file branch with the numpy-array mismatch guard, then clarify the intended behavior when both arguments are supplied. Done means the conflicting inputs are handled explicitly and the related documentation no longer leaves precedence ambiguous.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
database, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.