apache / apache/datafusion

Support MERGE INTO

Open
#20,746 2 comments 8 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The goal is to support MERGE INTO SQL statements in DataFusion so that downstream table providers (specifically Iceberg via iceberg-rust) can implement merge logic. The iceberg-rust repo (feature/merge-into branch) already has a merge_into function on its DataFusion table provider that expects DataFusion to parse MERGE INTO SQL and invoke a merge_into hook on TableProvider.

Example SQL:
```
MERGE INTO target_table t
USING source_table s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.value = s.value
WHEN NOT MATCHED THEN INSERT (id, value) VALUES (s.id, s.value)
```

### Describe the solution you'd like

The implementation follows the same pattern as UPDATE/DELETE DML hooks (PR#19142). Reuse the existing DmlStatement logical plan node with a new WriteOp::MergeInto(MergeIntoOp) variant. The merge-specific data (ON condition, WHEN clauses) is carried in MergeIntoOp. The DmlStatement.input field holds the source plan (USING clause), and DmlStatement.target holds the target table.

sqlparser v0.61.0 already parses Statement::Merge — no parser work needed.

The following tasks are already implemented in the [PoC branch](https://github.com/wirybeaver/datafusion/tree/feature/mergeinto) with 3 commits. Plan to raise PRs one by another as the fork repo doesn't support stacking PRs.
- [x] https://github.com/apache/datafusion/pull/20763
- [x] Add merge_into hook to TableProvider trait; Add SQL planner (merge_to_plan) + physical planner dispatch https://github.com/apache/datafusion/pull/22988
- [ ] Fix preserve MERGE target qualifier bindings https://github.com/apache/datafusion/pull/24429
- [ ] Add Memtable support, explain and docs https://github.com/apache/datafusion/pull/24195
- [x] Support scoped aliases in MERGE subqueries https://github.com/wirybeaver/datafusion/pull/4 Won't do. Replaced with #24429

Contributor guide

Open the contributing guide

Research direction

Start with the PoC branch and the existing UPDATE/DELETE DML hooks, then review DmlStatement, WriteOp, MergeIntoOp, TableProvider, and the SQL and physical planner work in PRs #20763 and #22988. Check the remaining target-qualifier task in #24429 and the MemTable, explain, and documentation work in #24195; done means the outstanding MERGE INTO support is covered without the already-rejected scoped-alias task.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.