apache / apache/datafusion-comet

Add a native Iceberg write benchmark

Open
#5,647 0 comments 0 reactions 0 assignees View on GitHub
area:Iceberg area:writer enhancement requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

The only performance numbers for native Iceberg writes are the ad hoc measurements in the #5361 description (2.9x unpartitioned, 5.5x partitioned fanout, from a harness in a personal fork). There is no benchmark in the repository, so a regression in the writer, in the shuffle feeding it, or in the JVM metrics rebuild would not be noticed, and the partitioned number was measured with `write.distribution-mode=none` rather than the default plan shape.

### Describe the potential solution

Add a Spark benchmark (the `org.apache.spark.sql.benchmark` framework used by `CometReadBenchmark` and the shuffle benchmark) that measures `INSERT INTO ... SELECT` into an Iceberg table for:

- unpartitioned
- partitioned with the default hash distribution (clustered writer, native shuffle and sort on the partition transforms after #5635)
- partitioned with `write.spark.fanout.enabled=true`
- a copy-on-write `DELETE` (rewrite of affected files)

Each case runs Spark, Comet scan only, and Comet scan plus native write, so the writer's contribution is isolated the way the #5361 table did. Wire it into the benchmark targets in the `Makefile` and document how to run it. Follow-on: run it on the EC2 benchmark runner and keep results with the other benchmark files.

Keep the Spark benchmark framework's warmup and minimum-time floors in mind when sizing the data (see the shuffle benchmark work in #5388 for how that dominated wall clock).

### Additional context

Part of the native Iceberg writes epic, #5649. Related: #5361, #3595 (fused sort plus write, which this benchmark would measure).

Contributor guide

Open the contributing guide

Research direction

Start with the org.apache.spark.sql.benchmark framework, CometReadBenchmark, and the shuffle benchmark work in #5388. Identify the benchmark targets in the Makefile and determine how the four Iceberg write cases compare Spark, Comet scan, and native write. Done means the benchmark runs with the framework's warmup and minimum-time floors, is wired into Make targets, and includes instructions for running it.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.