apache / apache/iceberg-python
Why non-atomic delete and upsert?
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 581
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 78
Description
### Question
The implementation of the `delete` and `upsert` operation make an deliberate choice to create multiple snapshots, breaking the atomicity expectation.
For example, if I start a `delete` operation to remove some entries, I do not expect consumers of the data to see a snapshot with some but not all of the entries gone.
Given that the implementation choice looks very deliberate, I'm curious about the rationale for it?
It seems that other clients of the library expect atomic operation as well, for example https://github.com/apache/iceberg-python/issues/3076
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points; begin by locating the delete and upsert implementations in PyIceberg and reviewing how they create snapshots. Determine whether the observed multiple-snapshot behavior is intentional, then document the rationale or define the required atomic behavior before making a change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100