SQL UPDATE Support for DataFusion Integration
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
### What's the feature are you trying to implement?
Implement SQL UPDATE functionality for iceberg-datafusion integration, enabling row-level updates with WHERE clause filtering. This completes the essential DML operations alongside INSERT and the upcoming `MERGE INTO` support https://github.com/apache/iceberg-rust/issues/2201.
Refer to the [insert_into support](https://github.com/apache/iceberg-rust/issues/1540)
**SQL Example**
```
UPDATE orders
SET status = 'shipped', shipped_date = current_date()
WHERE status = 'pending' AND payment_confirmed = true;
```
**Overall Architecture**
```
UPDATE table SET col1 = val1, col2 = val2 WHERE condition
↓
TableProvider::update() [NEW]
↓
IcebergTableScan (with WHERE filters)
↓
[Optional if partitioned: Project partition + Repartition + Sort for partitioned tables]
↓
IcebergUpdateWriteExec [NEW] - Apply assignments, write new files, track deleted
files
↓
CoalescePartitionsExec (reuse existing)
↓
IcebergUpdateCommitExec [NEW] - Commit via RowDelta transaction
↓
RecordBatch(count: UInt64)
```
*Strategy: Copy-on-Write (COW)*
1. Scan table with WHERE filters to find matching rows
2. Apply UPDATE assignments (evaluate expressions)
3. Write modified rows to new data files
4. Mark original files as deleted
5. Commit atomically with RowDelta (add new files + remove old files)
The following tasks are already completed on the [PoC branch with 6 commits](https://github.com/wirybeaver/iceberg-rust/tree/feature/update). Will raise formal PRs one after another as the fork repo doesn't support stacking PRs.
- [x] https://github.com/apache/iceberg-rust/pull/2203, shared with `MERGE INTO`
- [ ] IcebergUpdateWriteExec - write phase with conditional assignment application
- [ ] Add performance optimizations: Partition-aware updates (only scan/rewrite affected partitions) + File-level filtering using manifest statistics
- [ ] IcebergUpdateCommitExec - commit phase using RowDelta transaction
- [ ] TableProvider::update() trait implementation (Datafusion Hook)
- [ ] Integration tests and validation
### Willingness to contribute
I can contribute to this feature independently
Contributor guide
Research direction
Start by reviewing the PoC branch and the existing insert_into support, then follow the TableProvider::update() hook and the planned IcebergUpdateWriteExec and IcebergUpdateCommitExec components. Done means completing the write and RowDelta commit phases, adding the trait implementation, and passing the integration tests and validation described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100