lance-format / lance-format/lance

perf: opening an IVF index reads its two global buffers one after the other

Open
#9,377 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

IvfQuantizationStorage::try_new_with_remapper in rust/lance-index/src/vector/storage.rs reads two independent global buffers in sequence:

let ivf_bytes = reader.read_global_buffer(ivf_pos).await?;
let ivf = IvfModel::try_from(pb::Ivf::decode(ivf_bytes)?)?;
// ... parse the quantizer metadata JSON ...
if let Some(pos) = metadata.buffer_index() {
    let bytes = reader.read_global_buffer(pos).await?;
    metadata.parse_buffer(bytes)?;
}

The IVF protobuf holds the centroids; the second buffer holds whatever model the quantizer stores out of line, such as the PQ codebook or the RaBitQ rotation. Both positions are already in the file schema metadata before either read starts, so nothing forces the second read to wait for the first. Every cold index open pays an extra round trip for that, and since the IVF buffer runs to megabytes at a few thousand partitions, the transfers serialize as well.

Neither buffer is served from the retained tail in practice: on an IVF_PQ index with 16 partitions and PQBuildParams::new(4, 8) the two reads move about 32 KB from the store, well past the tail block.

I measured the serialization directly rather than inferring it. Wrapping the object store so every read is held open for 25 ms and recording how many are in flight, opening the storage of that index shows one read in flight at a time; issuing the two reads as a pair shows two.

Expected behavior

Issue both global buffer reads together and await them as a pair, since the second one's position is known before the first one is needed.

Lance version

13.0.0-beta.4 (main)

Language binding

Rust

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance-index/src/vector/storage.rs at IvfQuantizationStorage::try_new_with_remapper, and trace how the IVF and quantizer metadata buffers are located and parsed. Change the opening flow so both known global-buffer reads are awaited as a pair, then verify that IVF and quantizer metadata parsing and error handling still complete correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.