lance-format / lance-format/lance
bug: skip_transpose is ignored for IVF_HNSW_PQ
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
VectorIndexParams::skip_transpose decides whether PQ codes are stored transposed. build_vector_index_impl in rust/lance/src/index/vector.rs passes it to the builder for IvfPq and IvfRq, and for the incremental and optimize paths:
.with_transpose(!params.skip_transpose)
The IvfHnswPq arm does not, so its builder keeps the default and writes transposed codes whatever the caller asked for. IVF_HNSW_PQ has PQ storage like IVF_PQ, and the builder already handles the untransposed case generically (it un-transposes before writing and records transposed: false in the storage metadata, which readers honor), so there is nothing about the HNSW sub-index that makes the flag inapplicable. It is just not wired.
The result is visible in the index: with skip_transpose(true), an IVF_PQ index records transposed: false and an IVF_HNSW_PQ index records transposed: true.
Expected behavior
Pass the flag to the IVF_HNSW_PQ builder as well, so the stored layout matches what the caller asked for.
Lance version
13.0.0-beta.4 (main)
Language binding
Rust
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance/src/index/vector.rs at build_vector_index_impl and compare the IvfHnswPq arm with the IvfPq and IvfRq arms. Verify that skip_transpose is passed to the IVF_HNSW_PQ builder and that an index built with skip_transpose(true) records transposed: false in its storage metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100