apache / apache/datafusion-comet
Iceberg native scan claims schemes it cannot execute; three scheme lists disagree
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
CometScanRule.validateIcebergFileScanTasks checks Iceberg file schemes against a hardcoded set (file, s3, s3a, gs, gcs, oss, abfss, abfs, wasbs, wasb at CometScanRule.scala:1082), and its fallback message repeats that list to users. But the Iceberg execution path resolves storage through its own factory (iceberg_scan.rs storage_factory_for), which supports only file, s3, s3a, gs, oss via OpenDAL. And the JNI probe every other scheme gate delegates to (isNativelyReadableScheme → object_store::ObjectStoreScheme::parse) recognizes a third, different set.
Net effect: an Iceberg table under gcs, abfs, abfss, wasb, or wasbs passes validation, gets claimed, and every task dies with CometNativeException: Unsupported storage scheme: ... — while stock Spark reads the same table fine. Meanwhile oss actually works at execution despite the JNI probe saying it's unsupported, so the three lists disagree in both directions.
### Steps to reproduce
No cloud account needed — back the scheme with local disk the way ParquetReadFromFakeHadoopFsSuite does:
// register a local-backed filesystem for the wasb scheme
spark.conf.set("spark.hadoop.fs.wasb.impl", "org.apache.hadoop.fs.RawLocalFileSystem")
// hadoop catalog with a wasb warehouse
// spark.sql.catalog.hadoop_catalog = org.apache.iceberg.spark.SparkCatalog
// spark.sql.catalog.hadoop_catalog.type = hadoop
// spark.sql.catalog.hadoop_catalog.warehouse = wasb://fake-container/tmp/warehouse
sql("CREATE TABLE hadoop_catalog.t (id INT, s STRING) USING iceberg")
sql("INSERT INTO hadoop_catalog.t VALUES (1,'a'),(2,'b'),(3,'c')")
sql("SELECT * FROM hadoop_catalog.t").collect()
With spark.comet.enabled=false: [1,a],[2,b],[3,c].
With Comet + spark.comet.scan.icebergNative.enabled=true (plan shows CometIcebergNativeScan ... wasb://fake-container/...): every task fails with
org.apache.comet.CometNativeException: Unsupported storage scheme: wasb
(Ensure spark.comet.libhdfs.schemes doesn't include wasb, since that conf routes schemes elsewhere.)
### Expected behavior
Schemes the Iceberg native path can't execute should be declined at plan time so the scan falls back to Spark, and the fallback message should list what's actually supported. One source of truth: the validator should ask the same layer that executes — either expose the Iceberg storage factory's supported set through the existing JNI probe pattern, or derive the check from it directly — rather than maintaining a third hand-written list.
### Additional context
The claimed-but-fails set on this path is {gcs, abfs, abfss, wasb, wasbs}; oss works at execution but is rejected by the generic JNI probe, so aligning the lists fixes both directions. Same drift class as the scheme/config gates reworked in #5365.
Contributor guide
Research direction
Start at CometScanRule.scala:1082 and compare its scheme validation with iceberg_scan.rs storage_factory_for and isNativelyReadableScheme. Reproduce the wasb case using the local-backed filesystem setup from ParquetReadFromFakeHadoopFsSuite, then align validation and fallback reporting with the schemes the Iceberg execution path actually supports. Done means unsupported schemes fall back to Spark and supported schemes, including oss, are not rejected by the gate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala, spark
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100