apache / apache/datafusion-comet

[FEATURE] Native scan support for VariantType columns (Iceberg + Spark 4.0)

Open
#4,295 4 comments 1 reaction 1 assignee Claimed by @peterxcli View on GitHub
area:scan enhancement priority:high spark 4.0
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

Today, Comet falls back to JVM Spark for any query touching a VariantType column. This eliminates Comet's acceleration for the entire query, including unrelated operators that could otherwise run natively. This issue tracks adding native execution for Variant column scans, both for plain Parquet and Iceberg tables, by consuming new Variant primitives from arrow-rs and iceberg-rust.

### Describe the potential solution

```
spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala (line 729):

case s: StructType if isVariantStruct(s) =>
fallbackReasons +=
s"Unsupported $name of type VariantType (shredded; not supported by $scanImpl scan)"
false
```

### Additional context

Follow-ups

• Comet support for parse_json / to_variant (write side)
• Comet support for schema_of_variant (metadata operation)
• Predicate pushdown on variant_get(...) = literal into Parquet's column index
• Variant in shuffle / spill paths (separate issue)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.