lance-format / lance-format/lance

Support safe concurrent publication of derived columns in the same fragment

Open
#9,043 0 comments 0 reactions 1 assignee View on GitHub

@Xuanwo is already working on this.

Since Sep 10, 2026.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Two tasks can read the same source column and populate independent output columns in the same fragment:

source version V
task A: y = f(x)
task B: z = g(x)

Both results should be publishable when their dependencies remain valid. Publishing y should not invalidate the staged values for z, and publishing z must preserve y and reject results whose source inputs changed.

This issue tracks that end-to-end transaction contract, combining the cell preconditions in #8976 with column-update application and rebasing. It does not propose another precondition API or duplicate the Update migration in #6898; those are the related building blocks. It supersedes the narrower approach in #9040, which disables automatic rebasing but leaves callers responsible for validating dependencies and rebuilding fragment metadata.

Current gap

In Lance 862de0bbcf43089c743bb81ca740d81ab6bda6a2, Update rebasing can reject changes to the same fragment even when different columns were updated. This conservatively protects some read-compute-write operations, but prevents independent column publications from composing.

Simply allowing disjoint writes is insufficient. For example, a task computes y from x, then an overlay changes x before publication. RewriteColumns checks an overlay against the fields it writes; it does not carry the task's declared reads of x. Checking those dependencies outside the commit loop leaves a race if the transaction is subsequently rebased. This is a code-path example, not a separately executed reproducer attached to this issue.

Relevant code: Update conflict checks, transaction representation.

Desired behavior

The caller declares the original source version, the cells and structural/schema state on which its staged values depend, and the columns it intends to replace. Lance checks those preconditions inside the commit/rebase loop and applies the column changes to the latest manifest, preserving unrelated data files and overlays. A concurrent winner causes another check against the same original dependencies, not a silent reset of those dependencies to the newer version.

Dependencies must include copied values, not only expression inputs: if a staged output file includes values copied from rows outside a filter, those output cells are part of the read dependency. Application metadata that determines the computation also needs an appropriate precondition when it can change concurrently. Lance should enforce generic preconditions; function interpretation and recomputation remain the caller's responsibility.

The initial scope can use physical row addresses as in #8976. If a concurrent rewrite makes those addresses invalid, rejecting the staged result is sufficient; transparent remapping through compaction is not required for this case.

Acceptance cases
  • Two transactions read the same x and write disjoint y and z in one fragment. Both commit without recomputation, and both outputs remain correct regardless of commit order.
  • The same behavior holds when the columns originally share one base file and when the batch spans multiple fragments.
  • A concurrent input or filter-dependency change, including an overlay between validation and publication, rejects the stale result.
  • Conflicting writes to output/assignment cells are detected. Untouched output values copied into a staged file cannot overwrite a concurrent update.
  • Fragment removal, row relocation, and incompatible schema/dependency-metadata changes cannot bypass preconditions.
  • Safe rebasing preserves the existing contracts for deletions, overlays, index coverage, and row-version metadata.
  • A lost successful commit response can be recovered without publishing the same logical update twice.

Related work: #8976 (cell preconditions), #6898 (Update translation and conflict resolution), and #8644 (Transaction V2 actions). The implementation should compose with those efforts and test the combined read-dependency and write-application boundary.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.