lance-format / lance-format/lance
Top-level indices part 2: empty vector indices
@wjones127 is already working on this.
Since May 8, 2026.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Follow up to #3940
We are trying to make it possible to keep indices even though we don't have enough data to actually train them. In an earlier PR, we are adding a train: bool parameter to let the user choose whether they want to defer training an index. If train=False, then we just create an empty index and it will show num_indexed_rows: 0 in stats. They can later call optimize_indices() to do the actual training.
Vector indices are more complex because of IVF partitions, since we need a minimum amount of data to create them.
Here is the behavior I think we want:
- Can pass
train=Falseto create an empty index. No IVF or PQ will be trained. Will save metadata as empty array. - If you call
create_index(..., train=True)oroptimize_indices():- < 256 non-null vectors -> same as
train=False - 256 <= number of non-null vectors <
num_partitions * 256=> train on smaller number of partitions - number of non-null vectors >=
num_partitions * 256-> train full index
- < 256 non-null vectors -> same as
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.