lance-format / lance-format/lance
Bitmap index should support spilling
Open
@jtuglu1 is already working on this.
Since Oct 14, 2025.
A-index
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently the bitmap index just accumulates everything into a hashmap during training. If there are many unique values this can easily lead to OOM. We should spill on occasion to prevent this.
Example RAM-hungry operation:
import pyarrow as pa
import lance
import shutil
shutil.rmtree("/tmp/bitmap.lance", ignore_errors=True)
print("Writing data")
ds = lance.write_dataset(pa.table({"x": range(1_000_000_000)}), "/tmp/bitmap.lance")
print("Creating index")
ds.create_scalar_index("x", "BITMAP")
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.