apache / apache/datafusion-comet
[FEATURE] Native scan support for VariantType columns (Iceberg + Spark 4.0)
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
Today, Comet falls back to JVM Spark for any query touching a VariantType column. This eliminates Comet's acceleration for the entire query, including unrelated operators that could otherwise run natively. This issue tracks adding native execution for Variant column scans, both for plain Parquet and Iceberg tables, by consuming new Variant primitives from arrow-rs and iceberg-rust.
### Describe the potential solution
```
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala (line 729):
case s: StructType if isVariantStruct(s) =>
fallbackReasons +=
s"Unsupported $name of type VariantType (shredded; not supported by $scanImpl scan)"
false
```
### Additional context
Follow-ups
• Comet support for parse_json / to_variant (write side)
• Comet support for schema_of_variant (metadata operation)
• Predicate pushdown on variant_get(...) = literal into Parquet's column index
• Variant in shuffle / spill paths (separate issue)
Contributor guide
Assessment
This issue has not been assessed yet.