lance-format / lance-format/lance

cleanup_old_versions can delete files held by long-running readers

Open
#6,607 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

A Dataset handle at an older version (via checkout_version) has no way to signal cleanup_old_versions that it's in use. If cleanup runs while a reader is mid-scan, the manifest and data files for that version can be deleted out from under it, and the scan fails with object-store NotFound errors.

Reproduce
  1. Open a dataset and checkout_version(N) where N is older than lance.auto_cleanup.older_than.
  2. Start a scan. While the scan is running, run cleanup_old_versions with a before_version that includes N (or let auto-cleanup fire on the next commit).
  3. The scan fails mid-stream with NotFound from the object store.
Current workarounds

Create a temporary tag before the read and delete it after. Works, but every reader has to cooperate and it gets awkward on auto-cleanup-per-commit deployments — cleanup runs on every write.

What I'd want

A short-lived, renewable signal that a reader can publish for the version it's using: cleanup treats the version as retained while the signal is live, and a crashed reader doesn't pin the version forever. Something like an advisory lease with a TTL.

Happy to put up a PR if this direction sounds reasonable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cleanup_old_versions and checkout_version entry points described in the issue, then trace how an older-version Dataset scan accesses its manifest and data files. Define how a short-lived renewable reader signal should interact with before_version and automatic cleanup, and verify that an active scan remains readable while expired or crashed-reader signals do not retain versions forever.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.