[Bug] Data evolution self-merge can cross rollback snapshot lineage
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Description
Data evolution self-merge validates staged row-ID partial updates using the base snapshot ID, but not the snapshot UUID or lineage. Both the current Spark implementation and the PyPaimon Ray implementation in #9339 treat snapshot IDs as sufficient identity.
A rollback can delete newer snapshots, after which new commits reuse the same numeric snapshot IDs. A staged update from the old snapshot can therefore be accepted against a different replacement snapshot with the same ID (an ABA problem).
### Reproduction
This has been reproduced with Spark 3.5:
1. Enable row tracking and data evolution.
2. Create snapshot 1 containing `(id=1, b=10)`.
3. Create snapshot 2 containing `(id=2, b=20)`.
4. Stage a self-merge partial update `b = b + 1` for `id=2`, based on the original snapshot 2.
5. Roll back the table to snapshot 1.
6. Insert `(id=30, b=300)`, recreating snapshot ID 2 with a different UUID and reusing the relevant row ID.
7. Commit the old staged update.
Expected: `(id=30, b=300)` remains unchanged, or the old commit fails closed.
Actual: the commit succeeds and produces `(id=30, b=21)`. The old update is silently applied to the replacement row.
### Trigger window and impact
The trigger window is narrow: a self-merge must remain staged across a rollback, snapshot IDs must be recreated, and row-ID ranges must overlap. However, the impact is a silent wrong-row update rather than a clean commit failure.
### Follow-up scope
Fix Spark and Python together so that their behavior remains aligned:
- fail closed when `latest snapshot ID < base snapshot ID`;
- detect equal snapshot IDs with different snapshot UUIDs;
- prevent rollback snapshot-ID reuse from accepting staged updates from an old lineage;
- decide how the base snapshot UUID is carried through self-merge staging and commit conflict detection;
- add equivalent Spark and Python regression tests for `latest < base`, same-ID/different-UUID, and rollback ABA.
PR #9339 intentionally does not address this lineage problem. It remains aligned with the current Spark behavior and is scoped to forward snapshot rebase, conflict retry, and safe abort/temporary-file cleanup.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the Spark 3.5 rollback ABA reproduction, then trace base snapshot identity through Spark self-merge staging and commit conflict detection, followed by the PyPaimon Ray implementation. Add equivalent regression coverage for latest < base, same-ID/different-UUID, and rollback ABA cases. Done means old-lineage updates fail closed or leave replacement rows unchanged in both implementations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100