lance-format / lance-format/lance
cleanup_old_versions can delete files held by long-running readers
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Description
A Dataset handle at an older version (via checkout_version) has no way to signal cleanup_old_versions that it's in use. If cleanup runs while a reader is mid-scan, the manifest and data files for that version can be deleted out from under it, and the scan fails with object-store NotFound errors.
Reproduce
- Open a dataset and
checkout_version(N)where N is older thanlance.auto_cleanup.older_than. - Start a scan. While the scan is running, run
cleanup_old_versionswith abefore_versionthat includes N (or let auto-cleanup fire on the next commit). - The scan fails mid-stream with
NotFoundfrom the object store.
Current workarounds
Create a temporary tag before the read and delete it after. Works, but every reader has to cooperate and it gets awkward on auto-cleanup-per-commit deployments — cleanup runs on every write.
What I'd want
A short-lived, renewable signal that a reader can publish for the version it's using: cleanup treats the version as retained while the signal is live, and a crashed reader doesn't pin the version forever. Something like an advisory lease with a TTL.
Happy to put up a PR if this direction sounds reasonable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the cleanup_old_versions and checkout_version entry points described in the issue, then trace how an older-version Dataset scan accesses its manifest and data files. Define how a short-lived renewable reader signal should interact with before_version and automatic cleanup, and verify that an active scan remains readable while expired or crashed-reader signals do not retain versions forever.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100