lance-format / lance-format/lance

bug: skip_transpose is ignored for IVF_HNSW_PQ

Open Beginner friendly
#9,418 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

VectorIndexParams::skip_transpose decides whether PQ codes are stored transposed. build_vector_index_impl in rust/lance/src/index/vector.rs passes it to the builder for IvfPq and IvfRq, and for the incremental and optimize paths:

.with_transpose(!params.skip_transpose)

The IvfHnswPq arm does not, so its builder keeps the default and writes transposed codes whatever the caller asked for. IVF_HNSW_PQ has PQ storage like IVF_PQ, and the builder already handles the untransposed case generically (it un-transposes before writing and records transposed: false in the storage metadata, which readers honor), so there is nothing about the HNSW sub-index that makes the flag inapplicable. It is just not wired.

The result is visible in the index: with skip_transpose(true), an IVF_PQ index records transposed: false and an IVF_HNSW_PQ index records transposed: true.

Expected behavior

Pass the flag to the IVF_HNSW_PQ builder as well, so the stored layout matches what the caller asked for.

Lance version

13.0.0-beta.4 (main)

Language binding

Rust

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance/src/index/vector.rs at build_vector_index_impl and compare the IvfHnswPq arm with the IvfPq and IvfRq arms. Verify that skip_transpose is passed to the IVF_HNSW_PQ builder and that an index built with skip_transpose(true) records transposed: false in its storage metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
search
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.