lance-format / lance-format/lance
perf: opening an IVF index reads its two global buffers one after the other
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
IvfQuantizationStorage::try_new_with_remapper in rust/lance-index/src/vector/storage.rs reads two independent global buffers in sequence:
let ivf_bytes = reader.read_global_buffer(ivf_pos).await?;
let ivf = IvfModel::try_from(pb::Ivf::decode(ivf_bytes)?)?;
// ... parse the quantizer metadata JSON ...
if let Some(pos) = metadata.buffer_index() {
let bytes = reader.read_global_buffer(pos).await?;
metadata.parse_buffer(bytes)?;
}
The IVF protobuf holds the centroids; the second buffer holds whatever model the quantizer stores out of line, such as the PQ codebook or the RaBitQ rotation. Both positions are already in the file schema metadata before either read starts, so nothing forces the second read to wait for the first. Every cold index open pays an extra round trip for that, and since the IVF buffer runs to megabytes at a few thousand partitions, the transfers serialize as well.
Neither buffer is served from the retained tail in practice: on an IVF_PQ index with 16 partitions and PQBuildParams::new(4, 8) the two reads move about 32 KB from the store, well past the tail block.
I measured the serialization directly rather than inferring it. Wrapping the object store so every read is held open for 25 ms and recording how many are in flight, opening the storage of that index shows one read in flight at a time; issuing the two reads as a pair shows two.
Expected behavior
Issue both global buffer reads together and await them as a pair, since the second one's position is known before the first one is needed.
Lance version
13.0.0-beta.4 (main)
Language binding
Rust
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance-index/src/vector/storage.rs at IvfQuantizationStorage::try_new_with_remapper, and trace how the IVF and quantizer metadata buffers are located and parsed. Change the opening flow so both known global-buffer reads are awaited as a pair, then verify that IVF and quantizer metadata parsing and error handling still complete correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100