lance-format / lance-format/lance
Data file column tombstone cleanup
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
With the addition of https://github.com/lancedb/lance/pull/2639 it is now possible to introduce tombstoned columns with merge_insert. A user reported that this led to a surprising amount of disk usage since the files with tombstones are still "active" (and won't be cleaned up) but the tombstoned data is redundant.
We could examine what percentage of a file is tombstoned and, if this is large (configurable threshold), we can rewrite the file and remove the tombstoned columns.
This can be run as part of cleanup or potentially compaction / optimize.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing the cleanup and compaction/optimize paths that handle active files and tombstoned columns. Define how the configurable tombstone threshold is measured and verify that qualifying files are rewritten with redundant tombstoned columns removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100