lance-format / lance-format/lance

DataReplacement: ReplaceFragmentColumns action + translation + conflict resolution

Open
#6,842 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Parent PRD

Milestone: Action-based Transactions (UserOperation) — see milestone #11. Discussion: #5960. Design spike: #6448.

Re-scoped. Originally translation-only. The ReplaceFragmentColumns action that spike #6448 assigned to #6450 was never added to the Action enum, and conflict resolution is now migrated per-operation (framework: #6892). This issue is therefore the full DataReplacement vertical slice.

What to build

Operation::DataReplacement has no action translation and still uses a legacy build_manifest arm. Combined vertical slice — new action, translation, apply routing, conflict migration.

  • Define the ReplaceFragmentColumns action (spike §4 A4): payload { fragment_id, field_ids, new_data_files, updated_row_offsets }; implement apply and validate. validate enforces the same-layout-across-fragments rule that Operation::DataReplacement checks today, reading the current per-field file layout from &Manifest — no expected_layout_hash payload field.
  • Translate DataReplacement[ReplaceFragmentColumns*], one per DataReplacementGroup. The replaced field-set is inferred from the swapped DataFile. Translation is apply-time (needs &Manifest). Round-trip losslessly.
  • Route build_manifest through Action::apply and delete the legacy DataReplacement arm.
  • Implement reads() / writes() / rebase() for ReplaceFragmentColumns.
  • Add a property-based generator; port the legacy check_data_replacement_txn unit tests onto the new resolver; extend the differential matrix; fill design-doc §5/§6.

conflict_resolver.rs must not be modified — it remains the production path and differential oracle until cutover (#6454).

Acceptance criteria

  • ReplaceFragmentColumns action defined; apply + validate (same-layout rule) implemented
  • DataReplacement translated to ReplaceFragmentColumns actions; round-trip test passes
  • build_manifest routed through Action::apply; legacy DataReplacement arm deleted
  • reads/writes/rebase implemented for ReplaceFragmentColumns
  • Property-based generator added
  • Legacy check_data_replacement_txn unit tests ported and green on the new resolver
  • Differential matrix extended; green for all migrated operations
  • Existing data-replacement tests still pass
  • conflict_resolver.rs unmodified

Blocked by

  • #6892 (conflict-resolution framework)

User stories addressed

  • User story 5: single code path for applying operations
  • User story 6: single code path for conflict resolution
  • User story 9: old operations translated to actions
  • User story 11: lossless round-trip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the DataReplacement handling, build_manifest, Action::apply, and the legacy check_data_replacement_txn tests described in the issue, then read design spike #6448 sections §4, §5, and §6. Track the conflict-resolution framework in #6892 before beginning. Done means the listed action, translation, routing, resolver behavior, generators, tests, and differential matrix are complete while conflict_resolver.rs remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.