apache / apache/datafusion

[EPIC] Complete DML Support (MERGE, INSERT OVERWRITE, TRUNCATE)

Open
#19,617 6 comments 1 reaction 1 assignee Claimed by @Nachiket-Roy View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

With #19142 merged, DataFusion has TableProvider hooks for DELETE and UPDATE. This epic tracks the remaining DML statements needed for a complete SQL surface.

Currently missing:

- **MERGE INTO** - downstream projects have to decompose this themselves
- **INSERT OVERWRITE** - important for partition replacement workflows
- **TRUNCATE TABLE** - currently requires DELETE WHERE true

### Describe the solution you'd like

**MERGE statement:**

- sqlparser-rs already parses MERGE (Statement::Merge)
- Add LogicalPlan::Merge node
- Physical planning via decomposition to existing DELETE/UPDATE/INSERT hooks
- TableProvider::merge_into() for storage-native implementations

**INSERT OVERWRITE:**

- Extend insert_into() or add separate hook
- Support static and dynamic partition overwrite modes

**TRUNCATE:**

- Add TableProvider::truncate() hook
- More efficient than full table scan + delete

**ON CONFLICT (nice-to-have):**

- PostgreSQL-style INSERT ... ON CONFLICT DO UPDATE/NOTHING
- Lower priority- rounds out the DML surface

### Describe alternatives you've considered

Downstream projects can continue implementing MERGE decomposition themselves. Works, but duplicates planning logic across integrations and pushes SQL semantics outside the query engine.

### Additional context

Related issues for context:

- #19142
- #12406
- #16959

Happy to take this on if there's interest!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.