apache / apache/datafusion-comet
Support Iceberg `SparkCopyOnWriteScan` in `IcebergReflection.ICEBERG_SCAN_CLASSES`
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
When running Copy-On-Write (CoW) table operations such as MERGE INTO, UPDATE, or DELETE against Iceberg tables in Spark Iceberg's Spark planning extension creates a
SparkCopyOnWriteScan (org.apache.iceberg.spark.source.SparkCopyOnWriteScan) for scanning the target table data files.
Found the gap while testing native iceberg writes #4487
Currently, IcebergReflection.ICEBERG_SCAN_CLASSES only includes:
• org.apache.iceberg.spark.source.SparkBatchQueryScan
• org.apache.iceberg.spark.source.SparkStagedScan
Because SparkCopyOnWriteScan is missing from ICEBERG_SCAN_CLASSES, IcebergReflection.isIcebergScanClass(...) returns false when CometScanRule evaluates a physical plan
containing a SparkCopyOnWriteScan. As a result, Comet fails to recognize SparkCopyOnWriteScan as a supported Iceberg scan and falls back to JVM Spark execution for the scan
phase of Copy-On-Write queries.
### Describe the potential solution
Add SparkCopyOnWriteScan ("org.apache.iceberg.spark.source.SparkCopyOnWriteScan") to IcebergReflection.ClassNames and include it in IcebergReflection.ICEBERG_SCAN_CLASSES.
// In IcebergReflection.scala
object ClassNames {
...
val SPARK_COPY_ON_WRITE_SCAN = "org.apache.iceberg.spark.source.SparkCopyOnWriteScan"
}
val ICEBERG_SCAN_CLASSES: Set[String] =
Set(
ClassNames.SPARK_BATCH_QUERY_SCAN,
ClassNames.SPARK_STAGED_SCAN,
ClassNames.SPARK_COPY_ON_WRITE_SCAN)
This enables CometScanRule to recognize SparkCopyOnWriteScan instances and convert them to native Comet scans during Iceberg Copy-On-Write MERGE, UPDATE, and DELETE
operations.
### Additional context
SparkCopyOnWriteScan extends Iceberg's standard Spark scan primitives (SparkBatchQueryScan / SparkScan) and behaves identically with respect to file tasks, table schemas, and
filter pushdowns during data file reads.
Contributor guide
Research direction
Open IcebergReflection.scala and trace how CometScanRule uses ICEBERG_SCAN_CLASSES to identify Iceberg scans. Add recognition for org.apache.iceberg.spark.source.SparkCopyOnWriteScan, then verify that Copy-On-Write MERGE, UPDATE, and DELETE scans are recognized for native Comet conversion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100