lance-format / lance-format/lance
Track encoded file concatenation and column slices
Open
@Xuanwo is already working on this.
Since Aug 21, 2026.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Goal
Land a single low-level primitive for concatenating compatible encoded Lance files and FileFragment APIs for independently writing and combining complete physical-row slices of existing columns.
Context
- Proposal: https://github.com/lance-format/lance/discussions/8615
- Preparatory API rename: https://github.com/lance-format/lance/pull/8622
- Rust implementation: https://github.com/lance-format/lance/pull/8660
Contract
- Compatible current-version files concatenate encoded page payloads without decode and re-encode.
- Unsupported layouts use the existing ordered decode/re-encode fallback; corrupt or missing inputs remain errors.
- Column slices are immutable, fragment-local physical-row intervals with exact field and source-snapshot identity.
- Publication requires complete, gap-free, non-overlapping fragment coverage.
- Compaction reuses the same encoded-file primitive without adopting the column-slice lifecycle.
- Python
ColumnSlicevalues support a versioned bytes round trip and pickling.
Out of scope
- Task planning, scheduling, retries, progress, recovery, or distributed commit coordination.
- A new manifest layout or transaction operation.
- Caller-owned staged-file retention and cleanup.
Completion gates
- https://github.com/lance-format/lance/pull/8660 is merged.
- Python serialization and fragment bindings implement the proposal contract.
- Encoded concatenation, fallback, malformed-input, coverage, and compaction-reuse tests pass.
- Public API and ownership boundaries are documented.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.