lance-format / lance-format/lance
Thread MemoryPool through vector IVF index build (incl. shuffler batch sizing)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Thread the per-build MemoryPool through the vector IVF build (shuffle + per-partition sub-index construction), using try_grow returning Err as the spill signal.
Also size the TwoFileShuffler batch_size_bytes off the pool instead of the fixed 128 MB default. Total read requests scale as num_batches * num_partitions, so a larger batch (a large fraction of the available budget) materially cuts request count; sizing it from the MemoryPool lets it grow on big machines and shrink under pressure.
Part of #7301. Builds on the TwoFileShuffler peak-memory fix (#7299).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the vector IVF index build entry point and trace how the per-build MemoryPool reaches shuffle and per-partition sub-index construction. Inspect TwoFileShuffler batch sizing and the existing try_grow error path; done means the pool controls allocations and batch_size_bytes while spill signaling remains handled correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- search
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100