lance-format / lance-format/lance
DataReplacement: ReplaceFragmentColumns action + translation + conflict resolution
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Parent PRD
Milestone: Action-based Transactions (UserOperation) — see milestone #11. Discussion: #5960. Design spike: #6448.
Re-scoped. Originally translation-only. The
ReplaceFragmentColumnsaction that spike #6448 assigned to #6450 was never added to theActionenum, and conflict resolution is now migrated per-operation (framework: #6892). This issue is therefore the full DataReplacement vertical slice.
What to build
Operation::DataReplacement has no action translation and still uses a legacy build_manifest arm. Combined vertical slice — new action, translation, apply routing, conflict migration.
- Define the
ReplaceFragmentColumnsaction (spike §4 A4): payload{ fragment_id, field_ids, new_data_files, updated_row_offsets }; implementapplyandvalidate.validateenforces the same-layout-across-fragments rule thatOperation::DataReplacementchecks today, reading the current per-field file layout from&Manifest— noexpected_layout_hashpayload field. - Translate
DataReplacement→[ReplaceFragmentColumns*], one perDataReplacementGroup. The replaced field-set is inferred from the swappedDataFile. Translation is apply-time (needs&Manifest). Round-trip losslessly. - Route
build_manifestthroughAction::applyand delete the legacyDataReplacementarm. - Implement
reads()/writes()/rebase()forReplaceFragmentColumns. - Add a property-based generator; port the legacy
check_data_replacement_txnunit tests onto the new resolver; extend the differential matrix; fill design-doc §5/§6.
conflict_resolver.rs must not be modified — it remains the production path and differential oracle until cutover (#6454).
Acceptance criteria
-
ReplaceFragmentColumnsaction defined;apply+validate(same-layout rule) implemented -
DataReplacementtranslated toReplaceFragmentColumnsactions; round-trip test passes -
build_manifestrouted throughAction::apply; legacyDataReplacementarm deleted -
reads/writes/rebaseimplemented forReplaceFragmentColumns - Property-based generator added
- Legacy
check_data_replacement_txnunit tests ported and green on the new resolver - Differential matrix extended; green for all migrated operations
- Existing data-replacement tests still pass
-
conflict_resolver.rsunmodified
Blocked by
- #6892 (conflict-resolution framework)
User stories addressed
- User story 5: single code path for applying operations
- User story 6: single code path for conflict resolution
- User story 9: old operations translated to actions
- User story 11: lossless round-trip
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the DataReplacement handling, build_manifest, Action::apply, and the legacy check_data_replacement_txn tests described in the issue, then read design spike #6448 sections §4, §5, and §6. Track the conflict-resolution framework in #6892 before beginning. Done means the listed action, translation, routing, resolver behavior, generators, tests, and differential matrix are complete while conflict_resolver.rs remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100