apache / apache/datafusion-comet

Accelerate row-level MERGE / UPDATE / DELETE plans (MergeRowsExec, ReplaceDataExec, WriteDeltaExec)

Open
#5,122 4 comments 0 reactions 0 assignees View on GitHub
area:Iceberg area:writer enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Row-level `MERGE` / `UPDATE` / `DELETE` plans fall back to Spark entirely: `MergeRowsExec` (Spark 3.5+), `ReplaceDataExec`, `WriteDeltaExec`, and `InsertOnlyMergeExec` (Spark 4.2+).

CDC upsert into Iceberg is one of the most common modern ETL workloads. The expensive parts of a `MERGE` plan (the source/target join, filters, projections) are operators Comet already accelerates, but the merge tail forces the stage back to Spark, so in practice the whole statement runs without acceleration.

### Describe the potential solution

`MergeRowsExec` itself is close to a projection: it routes each joined row through matched / not-matched instruction lists that are ordinary Catalyst expressions, emitting updated, inserted, or deleted rows plus the row-operation column. That makes it a candidate for a native operator independent of native writes, which would at least keep the join and merge logic in one native stage before handing off to the JVM writer.

Full acceleration of `ReplaceDataExec` / `WriteDeltaExec` also needs native DataSource V2 writes, tracked in #5121.

### Additional context

Related: #5121 (DataSource V2 writes), #1625 (EPIC: native Parquet writes), #3756 (Iceberg feature matrix).

Contributor guide

Open the contributing guide

Research direction

Start with the Spark row-level plan entry points named in the issue: MergeRowsExec, ReplaceDataExec, WriteDeltaExec, and InsertOnlyMergeExec, then read related issues #5121, #1625, and #3756 for scope and existing constraints. Done means the applicable merge logic remains in a native stage with coverage for the intended row operations; native DataSource V2 writes are explicitly tracked separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala, spark
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.