lance-format / lance-format/lance

Track encoded file concatenation and column slices

Open
#8,695 0 comments 0 reactions 1 assignee View on GitHub

@Xuanwo is already working on this.

Since Aug 21, 2026.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Goal

Land a single low-level primitive for concatenating compatible encoded Lance files and FileFragment APIs for independently writing and combining complete physical-row slices of existing columns.

Context

Contract

  • Compatible current-version files concatenate encoded page payloads without decode and re-encode.
  • Unsupported layouts use the existing ordered decode/re-encode fallback; corrupt or missing inputs remain errors.
  • Column slices are immutable, fragment-local physical-row intervals with exact field and source-snapshot identity.
  • Publication requires complete, gap-free, non-overlapping fragment coverage.
  • Compaction reuses the same encoded-file primitive without adopting the column-slice lifecycle.
  • Python ColumnSlice values support a versioned bytes round trip and pickling.

Out of scope

  • Task planning, scheduling, retries, progress, recovery, or distributed commit coordination.
  • A new manifest layout or transaction operation.
  • Caller-owned staged-file retention and cleanup.

Completion gates

  • https://github.com/lance-format/lance/pull/8660 is merged.
  • Python serialization and fragment bindings implement the proposal contract.
  • Encoded concatenation, fallback, malformed-input, coverage, and compaction-reuse tests pass.
  • Public API and ownership boundaries are documented.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.