apache / apache/datafusion-comet

Support Iceberg `SparkCopyOnWriteScan` in `IcebergReflection.ICEBERG_SCAN_CLASSES`

Open Beginner friendly
#5,319 0 comments 0 reactions 0 assignees View on GitHub
area:scan enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

When running Copy-On-Write (CoW) table operations such as MERGE INTO, UPDATE, or DELETE against Iceberg tables in Spark Iceberg's Spark planning extension creates a
SparkCopyOnWriteScan (org.apache.iceberg.spark.source.SparkCopyOnWriteScan) for scanning the target table data files.

Found the gap while testing native iceberg writes #4487

Currently, IcebergReflection.ICEBERG_SCAN_CLASSES only includes:

• org.apache.iceberg.spark.source.SparkBatchQueryScan
• org.apache.iceberg.spark.source.SparkStagedScan

Because SparkCopyOnWriteScan is missing from ICEBERG_SCAN_CLASSES, IcebergReflection.isIcebergScanClass(...) returns false when CometScanRule evaluates a physical plan
containing a SparkCopyOnWriteScan. As a result, Comet fails to recognize SparkCopyOnWriteScan as a supported Iceberg scan and falls back to JVM Spark execution for the scan
phase of Copy-On-Write queries.

### Describe the potential solution

Add SparkCopyOnWriteScan ("org.apache.iceberg.spark.source.SparkCopyOnWriteScan") to IcebergReflection.ClassNames and include it in IcebergReflection.ICEBERG_SCAN_CLASSES.

// In IcebergReflection.scala
object ClassNames {
...
val SPARK_COPY_ON_WRITE_SCAN = "org.apache.iceberg.spark.source.SparkCopyOnWriteScan"
}

val ICEBERG_SCAN_CLASSES: Set[String] =
Set(
ClassNames.SPARK_BATCH_QUERY_SCAN,
ClassNames.SPARK_STAGED_SCAN,
ClassNames.SPARK_COPY_ON_WRITE_SCAN)

This enables CometScanRule to recognize SparkCopyOnWriteScan instances and convert them to native Comet scans during Iceberg Copy-On-Write MERGE, UPDATE, and DELETE
operations.

### Additional context

SparkCopyOnWriteScan extends Iceberg's standard Spark scan primitives (SparkBatchQueryScan / SparkScan) and behaves identically with respect to file tasks, table schemas, and
filter pushdowns during data file reads.

Contributor guide

Open the contributing guide

Research direction

Open IcebergReflection.scala and trace how CometScanRule uses ICEBERG_SCAN_CLASSES to identify Iceberg scans. Add recognition for org.apache.iceberg.spark.source.SparkCopyOnWriteScan, then verify that Copy-On-Write MERGE, UPDATE, and DELETE scans are recognized for native Comet conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, data-engineering
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.