apache / apache/datafusion-comet

Accelerate DataSource V2 writes (AppendDataExec, OverwriteByExpressionExec, OverwritePartitionsDynamicExec)

Open
#5,121 2 comments 0 reactions 0 assignees View on GitHub
area:Iceberg area:writer enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Comet accelerates DataSource V2 reads (Parquet, Iceberg via iceberg-rust, CSV) but all DataSource V2 writes fall back to Spark: `AppendDataExec`, `OverwriteByExpressionExec`, `OverwritePartitionsDynamicExec`, and `WriteToDataSourceV2Exec`.

For ETL pipelines this is the missing half of the job. A pipeline that scans, joins, and aggregates natively still pays a columnar-to-row transition into Spark's row-based writer at the end, and for Iceberg specifically we accelerate reads but not writes.

### Describe the potential solution

Suggested staging:

1. Start with Iceberg `AppendDataExec` (plain `INSERT INTO` / `append`), reusing the native Parquet write machinery from #1625 and coordinating with the iceberg-rust integration. #3595 covers the sort-merge/clustered writer path and would build on the same foundation.
2. `OverwriteByExpressionExec` and `OverwritePartitionsDynamicExec` (dynamic partition overwrite) next, since `INSERT OVERWRITE` is a standard ETL pattern.
3. Generic `WriteToDataSourceV2Exec` last, gated on the target format being one Comet can write natively.

As with the V1 native Parquet write work, this should be opt-in and experimental until commit protocol semantics are proven out.

### Additional context

Related: #1625 (EPIC: native Parquet writes), #3595 (native sort-merge writer for Iceberg clustered path), #3756 (Iceberg feature matrix). Row-level MERGE / UPDATE / DELETE plans are tracked separately since they layer on top of this.

Contributor guide

Open the contributing guide

Research direction

Start with the Iceberg AppendDataExec stage and read the native Parquet write work in #1625, along with the iceberg-rust integration context. Done means an opt-in experimental native path for Iceberg append writes, with commit protocol semantics addressed before expanding to overwrite and generic WriteToDataSourceV2Exec paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.