apache / apache/datafusion-comet

[Variant] Support native casts to and from VariantType

Open
#5,430 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Spark 4.x supports `CAST` / `TRY_CAST` between `VariantType` and a defined set of scalar and nested types, but Comet explicitly rejects every pair involving Variant in [`CometCast`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/comet/expressions/CometCast.scala#L175-L187).

Examples that therefore fall back include:

```sql
SELECT CAST(v AS STRING), TRY_CAST(v AS ARRAY) FROM t;
SELECT CAST(id AS VARIANT), CAST(array(1, 2) AS VARIANT) FROM t;
```

Spark admits casts from Variant through [`VariantGet.checkDataType`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L127-L142) and supports the reciprocal cast to Variant for scalar values and supported arrays. Structs and maps use `to_variant_object` instead. The runtime routes Variant casts through Spark's [Variant cast implementation](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1239-L1258), including [generated execution](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1343-L1366).

### Describe the potential solution

Add only Spark-supported Variant pairs to Comet's cast matrix and native cast implementation:

- reuse the extraction/conversion semantics from #5424 for casts from Variant, including nested targets once supported;
- encode casts to Variant with canonical `[value, metadata]` storage and the Variant result Field contract from #5425;
- preserve SQL NULL versus Variant JSON null, decimal, binary, date/timestamp, time-zone, overflow, and recursive null behavior;
- match `CAST` versus `TRY_CAST` error behavior; and
- keep casts that Spark does not admit unsupported. In particular, maps and structs must use `to_variant_object`, not a broader Variant cast.

Add focused parity and native-plan tests across supported scalar types, arrays and nested targets, SQL/JSON nulls, invalid conversions, `TRY_CAST`, aliases, and columns around a Variant-producing result. Spark 3.x compilation and behavior must remain unchanged through the version shim.

### Additional context

Related work: #4295, #5407, #5424, #5425, and the dedicated `to_variant_object` tracker.

Non-goals: comparisons and hash expressions (Spark rejects Variant for both), `to_variant_object`, parsing JSON, shuffle/spill, C2R, writing, and Python transport.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.