lance-format / lance-format/lance
Re-parameterize IVF index based on target partition size
@BubbleCal is already working on this.
Since Sep 2, 2025.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
We'd like to support the experience where users can set their desired index configuration before adding data. And do have their index configuration be sensible even as they add or delete rows. Currently, users set a fixed num_partitions value, which is only helpful within a certain range of row counts.
Instead, we should have them set a target_partition_size parameter, which can scale appropriately as they change their dataset size. optimize_indices should automatically handle retraining IVF depending when the num_partitions has drifted far enough from the ideal / requested value.
The default for target_partition_size should be 4096. That works well for datasets with fewer than 10 million rows. After that, we start to get too many partitions. So the calculation needs to be:
num_partitions = min( num_rows / target_partition_size, sqrt(num_rows), max_partitions )
Related issues
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.