apache / apache/datafusion-comet
[Variant] Support native casts to and from VariantType
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Spark 4.x supports `CAST` / `TRY_CAST` between `VariantType` and a defined set of scalar and nested types, but Comet explicitly rejects every pair involving Variant in [`CometCast`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala#L175-L187).
Examples that therefore fall back include:
```sql
SELECT CAST(v AS STRING), TRY_CAST(v AS ARRAY) FROM t;
SELECT CAST(id AS VARIANT), CAST(array(1, 2) AS VARIANT) FROM t;
```
Spark admits casts from Variant through [`VariantGet.checkDataType`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L127-L142) and supports the reciprocal cast to Variant for scalar values and supported arrays. Structs and maps use `to_variant_object` instead. The runtime routes Variant casts through Spark's [Variant cast implementation](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1239-L1258), including [generated execution](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1343-L1366).
### Describe the potential solution
Add only Spark-supported Variant pairs to Comet's cast matrix and native cast implementation:
- reuse the extraction/conversion semantics from #5424 for casts from Variant, including nested targets once supported;
- encode casts to Variant with canonical `[value, metadata]` storage and the Variant result Field contract from #5425;
- preserve SQL NULL versus Variant JSON null, decimal, binary, date/timestamp, time-zone, overflow, and recursive null behavior;
- match `CAST` versus `TRY_CAST` error behavior; and
- keep casts that Spark does not admit unsupported. In particular, maps and structs must use `to_variant_object`, not a broader Variant cast.
Add focused parity and native-plan tests across supported scalar types, arrays and nested targets, SQL/JSON nulls, invalid conversions, `TRY_CAST`, aliases, and columns around a Variant-producing result. Spark 3.x compilation and behavior must remain unchanged through the version shim.
### Additional context
Related work: #4295, #5407, #5424, #5425, and the dedicated `to_variant_object` tracker.
Non-goals: comparisons and hash expressions (Spark rejects Variant for both), `to_variant_object`, parsing JSON, shuffle/spill, C2R, writing, and Python transport.
Contributor guide
Assessment
This issue has not been assessed yet.