Support MERGE INTO
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
The goal is to support MERGE INTO SQL statements in DataFusion so that downstream table providers (specifically Iceberg via iceberg-rust) can implement merge logic. The iceberg-rust repo (feature/merge-into branch) already has a merge_into function on its DataFusion table provider that expects DataFusion to parse MERGE INTO SQL and invoke a merge_into hook on TableProvider.
Example SQL:
```
MERGE INTO target_table t
USING source_table s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.value = s.value
WHEN NOT MATCHED THEN INSERT (id, value) VALUES (s.id, s.value)
```
### Describe the solution you'd like
The implementation follows the same pattern as UPDATE/DELETE DML hooks (PR#19142). Reuse the existing DmlStatement logical plan node with a new WriteOp::MergeInto(MergeIntoOp) variant. The merge-specific data (ON condition, WHEN clauses) is carried in MergeIntoOp. The DmlStatement.input field holds the source plan (USING clause), and DmlStatement.target holds the target table.
sqlparser v0.61.0 already parses Statement::Merge — no parser work needed.
The following tasks are already implemented in the [PoC branch](https://github.com/wirybeaver/datafusion/tree/feature/mergeinto) with 3 commits. Plan to raise PRs one by another as the fork repo doesn't support stacking PRs.
- [x] https://github.com/apache/datafusion/pull/20763
- [x] Add merge_into hook to TableProvider trait; Add SQL planner (merge_to_plan) + physical planner dispatch https://github.com/apache/datafusion/pull/22988
- [ ] Fix preserve MERGE target qualifier bindings https://github.com/apache/datafusion/pull/24429
- [ ] Add Memtable support, explain and docs https://github.com/apache/datafusion/pull/24195
- [x] Support scoped aliases in MERGE subqueries https://github.com/wirybeaver/datafusion/pull/4 Won't do. Replaced with #24429
Contributor guide
Research direction
Start with the PoC branch and the existing UPDATE/DELETE DML hooks, then review DmlStatement, WriteOp, MergeIntoOp, TableProvider, and the SQL and physical planner work in PRs #20763 and #22988. Check the remaining target-qualifier task in #24429 and the MemTable, explain, and documentation work in #24195; done means the outstanding MERGE INTO support is covered without the already-rejected scoped-alias task.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100