apache / apache/iceberg-python

Why non-atomic delete and upsert?

Open
#3,577 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

### Question

The implementation of the `delete` and `upsert` operation make an deliberate choice to create multiple snapshots, breaking the atomicity expectation.

For example, if I start a `delete` operation to remove some entries, I do not expect consumers of the data to see a snapshot with some but not all of the entries gone.

Given that the implementation choice looks very deliberate, I'm curious about the rationale for it?

It seems that other clients of the library expect atomic operation as well, for example https://github.com/apache/iceberg-python/issues/3076

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points; begin by locating the delete and upsert implementations in PyIceberg and reviewing how they create snapshots. Determine whether the observed multiple-snapshot behavior is intentional, then document the rationale or define the required atomic behavior before making a change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.