apache / apache/iceberg-rust

SQL UPDATE Support for DataFusion Integration

Open
#2,205 5 comments 1 reaction 0 assignees View on GitHub
epic
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### What's the feature are you trying to implement?

Implement SQL UPDATE functionality for iceberg-datafusion integration, enabling row-level updates with WHERE clause filtering. This completes the essential DML operations alongside INSERT and the upcoming `MERGE INTO` support https://github.com/apache/iceberg-rust/issues/2201.

Refer to the [insert_into support](https://github.com/apache/iceberg-rust/issues/1540)

**SQL Example**
```
UPDATE orders
SET status = 'shipped', shipped_date = current_date()
WHERE status = 'pending' AND payment_confirmed = true;
```

**Overall Architecture**
```
UPDATE table SET col1 = val1, col2 = val2 WHERE condition

TableProvider::update() [NEW]

IcebergTableScan (with WHERE filters)

[Optional if partitioned: Project partition + Repartition + Sort for partitioned tables]

IcebergUpdateWriteExec [NEW] - Apply assignments, write new files, track deleted
files

CoalescePartitionsExec (reuse existing)

IcebergUpdateCommitExec [NEW] - Commit via RowDelta transaction

RecordBatch(count: UInt64)
```

*Strategy: Copy-on-Write (COW)*
1. Scan table with WHERE filters to find matching rows
2. Apply UPDATE assignments (evaluate expressions)
3. Write modified rows to new data files
4. Mark original files as deleted
5. Commit atomically with RowDelta (add new files + remove old files)

The following tasks are already completed on the [PoC branch with 6 commits](https://github.com/wirybeaver/iceberg-rust/tree/feature/update). Will raise formal PRs one after another as the fork repo doesn't support stacking PRs.
- [x] https://github.com/apache/iceberg-rust/pull/2203, shared with `MERGE INTO`
- [ ] IcebergUpdateWriteExec - write phase with conditional assignment application
- [ ] Add performance optimizations: Partition-aware updates (only scan/rewrite affected partitions) + File-level filtering using manifest statistics
- [ ] IcebergUpdateCommitExec - commit phase using RowDelta transaction
- [ ] TableProvider::update() trait implementation (Datafusion Hook)
- [ ] Integration tests and validation

### Willingness to contribute

I can contribute to this feature independently

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the PoC branch and the existing insert_into support, then follow the TableProvider::update() hook and the planned IcebergUpdateWriteExec and IcebergUpdateCommitExec components. Done means completing the write and RowDelta commit phases, adding the trait implementation, and passing the integration tests and validation described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.