lance-format / lance-format/lance

Action-level conflict resolution: framework + Append/Delete tracer bullet

Open
#6,892 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Parent PRD

Milestone: Action-based Transactions (UserOperation) — see milestone #11 for the full PRD.
Discussion: #5960. Conflict-resolution design spike: #6448.

Background

Spike #6448 produced two outputs: (a) the action catalog and (b) the action trait shape — a five-method contract (reads/writes/validate/rebase/apply) plus a driver loop. Output (a) was threaded into #6450–#6453; output (b) was never assigned to an implementation issue. As a result, build_manifest was migrated to Action::apply (9/13 ops, done under #6454), but conflict resolution still runs entirely through the legacy per-Operation check_*_txn matrix in rust/lance/src/io/commit/conflict_resolver.rs. The shipped Action::conflicts_with -> bool is the pre-spike shape and has no production callers.

Conflict resolution cannot be migrated one operation at a time in production: a conflict check is per pair of transactions, so during a transition a commit can hit a mixed (migrated, non-migrated) pair. The migration is therefore a parallel/shadow implementation, validated by a differential test matrix over the set of already-migrated operations (M×M), cutting over only once M = N.

This issue builds that framework and proves it end-to-end on Append + Delete.

What to build

  • Land the spike design doc at rust/lance-table/design/action_based_conflict_resolution.md (skeleton + #6448's resolved open questions; the §5/§6 conflict matrix is filled incrementally by the per-operation issues).
  • Implement the spike §7 trait on Action: reads() -> ManifestMask, writes() -> ManifestMask, async rebase(&mut self, current, concurrent); generalize validate across all actions. Define ManifestMask, TxnContext, ActionOutput.
  • Implement the transaction driver commit loop (mask-gated rebase, spike §7).
  • Implement a parallel conflict-resolution entry point behind a legacy-shaped adapter: same (other_transaction, other_version) -> Result<()> signature and rebased-transaction output as TransactionRebase::check_txn, so the eventual cutover is a one-line call-site swap. Not wired into the production commit path.
  • Implement reads/writes/rebase for the actions Append + Delete use: AddFragments, RemoveFragments, UpdateDeletionVector. Remaining actions get explicit unimplemented stubs, filled by their per-operation issue.
  • Handle Restore/Clone as top-level non-action transaction kinds in the driver.
  • Build the differential test harness + property-based transaction generator infrastructure; generators for Append and Delete.
  • Port the legacy Append/Delete conflict unit tests onto the new resolver; feed the same fixtures through the differential harness so unit-test expectations are cross-checked against the legacy oracle.

conflict_resolver.rs must not be modified — it remains the production path and the differential oracle until the final cutover (#6454).

Acceptance criteria

  • Design doc landed in-tree
  • reads/writes/validate/rebase/apply trait + ManifestMask/TxnContext/ActionOutput defined
  • Driver commit loop implemented
  • Parallel resolver entry point behind a legacy-shaped adapter
  • Differential harness + property-based generators for Append/Delete
  • M×M differential tests green for {Append, Delete}
  • Legacy Append/Delete conflict unit tests ported and green on the new resolver
  • conflict_resolver.rs unmodified

Blocked by

None — startable immediately.

Blocks

Per-operation conflict-resolution migration, then the cutover:

  • #6893 — metadata operations (UpdateConfig, UpdateBases, UpdateMemWalState, ReserveFragments)
  • #6894 — schema operations (Project, Merge)
  • #6895 — CreateIndex
  • #6896 — Overwrite (translation + conflict)
  • #6897 — Rewrite (translation + conflict)
  • #6898 — Update (translation + conflict)
  • #6842 — DataReplacement (translation + conflict)
  • #6454 — final cutover

User stories addressed

  • User story 6: single code path for conflict resolution

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with rust/lance-table/design/action_based_conflict_resolution.md and the §7 design from #6448, then inspect the existing transaction and action code around the conflict-resolution path. Build the parallel resolver without modifying rust/lance/src/io/commit/conflict_resolver.rs, and use the specified Append/Delete fixtures, generators, and differential tests as the completion check.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, distributed-systems, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.