apache / apache/datafusion-comet
Accelerate row-level MERGE / UPDATE / DELETE plans (MergeRowsExec, ReplaceDataExec, WriteDeltaExec)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Row-level `MERGE` / `UPDATE` / `DELETE` plans fall back to Spark entirely: `MergeRowsExec` (Spark 3.5+), `ReplaceDataExec`, `WriteDeltaExec`, and `InsertOnlyMergeExec` (Spark 4.2+).
CDC upsert into Iceberg is one of the most common modern ETL workloads. The expensive parts of a `MERGE` plan (the source/target join, filters, projections) are operators Comet already accelerates, but the merge tail forces the stage back to Spark, so in practice the whole statement runs without acceleration.
### Describe the potential solution
`MergeRowsExec` itself is close to a projection: it routes each joined row through matched / not-matched instruction lists that are ordinary Catalyst expressions, emitting updated, inserted, or deleted rows plus the row-operation column. That makes it a candidate for a native operator independent of native writes, which would at least keep the join and merge logic in one native stage before handing off to the JVM writer.
Full acceleration of `ReplaceDataExec` / `WriteDeltaExec` also needs native DataSource V2 writes, tracked in #5121.
### Additional context
Related: #5121 (DataSource V2 writes), #1625 (EPIC: native Parquet writes), #3756 (Iceberg feature matrix).
Contributor guide
Research direction
Start with the Spark row-level plan entry points named in the issue: MergeRowsExec, ReplaceDataExec, WriteDeltaExec, and InsertOnlyMergeExec, then read related issues #5121, #1625, and #3756 for scope and existing constraints. Done means the applicable merge logic remains in a native stage with coverage for the intended row operations; native DataSource V2 writes are explicitly tracked separately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala, spark
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100