lance-format / lance-format/lance
No Python API to prune frag-reuse generations, so deferred-remap users accumulate read-path debt forever
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Deferred-remap compaction (defer_index_remap=true) appends a new generation to the __lance_frag_reuse system index on every commit. Rust already has a complete and race-safe pruner, cleanup_frag_reuse_index in rust/lance/src/dataset/index/frag_reuse.rs, which drops every generation that all indexes have caught up to. However, this function is not exposed in Python and is never invoked automatically, so a Python pipeline that defers remaps accumulates generations without bound. Because the frag-reuse index is loaded on every vector index open, the open-time cost grows with compaction history even after every index has been rebuilt and no generation is needed anymore.
I propose exposing the existing function as LanceDataset.cleanup_frag_reuse_index() so it can be scheduled as a periodic maintenance call alongside cleanup_old_versions. A patch is available that follows the migrate_manifest_paths_v2 wrapper pattern and includes a test asserting that __lance_frag_reuse reports zero versions once all indexes have caught up and the cleanup runs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with cleanup_frag_reuse_index in rust/lance/src/dataset/index/frag_reuse.rs, then compare the Python wrapper for migrate_manifest_paths_v2. Add the LanceDataset cleanup entry point and a test verifying that __lance_frag_reuse reports zero versions after all indexes have caught up and cleanup runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- api, database
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100