lance-format / lance-format/lance
Building Bitmap indexes uses a large number of allocations
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Indexing 100MB of int64 data creates 1.4 million allocations.
Most of this seems to come from resizing containers within the RoaringBitmaps when calling insert.
RoaringBitmaps are designed to be efficiently built from data in ascending order. In theory, we should already be delivering data in order of row_address, so we might be able to speed this up easily by taking advantage of that.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at rust/lance-index/src/scalar/bitmap.rs around line 731 and inspect how insert resizes the RoaringBitmaps. Reproduce the 100MB int64 indexing case and measure allocations, then verify that exploiting ascending row_address input reduces allocations without changing bitmap results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100