lance-format / lance-format/lance
Track physical column layout optimization
Open
@Xuanwo is already working on this.
Since Aug 21, 2026.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Goal
Add an explicit physical-only maintenance operation that groups existing top-level fields into fewer standard per-fragment data files without changing logical rows or dataset semantics.
Context
Implementation and benchmark: https://github.com/lance-format/lance/pull/8614
Contract
- Callers choose top-level field groups based on their workload.
- Rewrites preserve logical values, fragment IDs, physical row order, deletion files, stable row IDs, and row-version metadata.
- A fixed snapshot cutoff materializes visible overlays while newer overlays remain authoritative.
- Disjoint field-level work can rebase safely; overlapping output conflicts fail.
- Index coverage for rewritten fields is invalidated conservatively.
- Legacy Blob, Blob v2, nested Blob v2, and external references use the existing safe rewrite machinery.
- The operation remains explicit policy and does not add an automatic scheduler.
Out of scope
- Automatic field grouping or workload learning.
- Changing logical row layout or row addresses.
- Immediate deletion of superseded provider objects.
- Automatic index rebuild scheduling.
Completion gates
- https://github.com/lance-format/lance/pull/8614 is rebased, passes CI, is reviewed, and is merged.
- Rewrite, overlay, conflict, index, Blob, and stable-row-ID coverage passes.
- Representative Local FS and object-storage benchmarks retain exact logical checksums.
- Read-amplification and rewrite-cost trade-offs are documented.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.