apache / apache/datafusion-comet
Accelerate DataSource V2 writes (AppendDataExec, OverwriteByExpressionExec, OverwritePartitionsDynamicExec)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Comet accelerates DataSource V2 reads (Parquet, Iceberg via iceberg-rust, CSV) but all DataSource V2 writes fall back to Spark: `AppendDataExec`, `OverwriteByExpressionExec`, `OverwritePartitionsDynamicExec`, and `WriteToDataSourceV2Exec`.
For ETL pipelines this is the missing half of the job. A pipeline that scans, joins, and aggregates natively still pays a columnar-to-row transition into Spark's row-based writer at the end, and for Iceberg specifically we accelerate reads but not writes.
### Describe the potential solution
Suggested staging:
1. Start with Iceberg `AppendDataExec` (plain `INSERT INTO` / `append`), reusing the native Parquet write machinery from #1625 and coordinating with the iceberg-rust integration. #3595 covers the sort-merge/clustered writer path and would build on the same foundation.
2. `OverwriteByExpressionExec` and `OverwritePartitionsDynamicExec` (dynamic partition overwrite) next, since `INSERT OVERWRITE` is a standard ETL pattern.
3. Generic `WriteToDataSourceV2Exec` last, gated on the target format being one Comet can write natively.
As with the V1 native Parquet write work, this should be opt-in and experimental until commit protocol semantics are proven out.
### Additional context
Related: #1625 (EPIC: native Parquet writes), #3595 (native sort-merge writer for Iceberg clustered path), #3756 (Iceberg feature matrix). Row-level MERGE / UPDATE / DELETE plans are tracked separately since they layer on top of this.
Contributor guide
Research direction
Start with the Iceberg AppendDataExec stage and read the native Parquet write work in #1625, along with the iceberg-rust integration context. Done means an opt-in experimental native path for Iceberg append writes, with commit protocol semantics addressed before expanding to overwrite and generic WriteToDataSourceV2Exec paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100