lance-format / lance-format/lance
Building FTS indices does a huge amount of allocations
Open
@Xuanwo is already working on this.
Since Dec 17, 2025.
A-index
performance
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Was building an FTS index on 100MB of text data, and got over 1 billion memory allocations. ^1
Here's a allocation flamegraph for indexing 1MB of data. It seems like a lot comes from serializes O(num_tokens) PostingListBuilders to Arrow RecordBatches.
There's some other small stuff like computing inverted_list_schema over and over again. That could be put in a LazyLock.
Without Positions
With Positions
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.